[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790376259650642-11032-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 22:44:20 dist-test-slave-5vdr krb5kdc[11039](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 22:44:20 dist-test-slave-5vdr krb5kdc[11039](info): set up 2 sockets
Sep 25 22:44:20 dist-test-slave-5vdr krb5kdc[11039](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260925 22:44:21.128937 11032 mini_kdc.cc:121] Time spent starting KDC: real 1.434s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790376259650642-11032-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790376259650642-11032-0/krb5kdc/test-user.keytab.
Sep 25 22:44:21 dist-test-slave-5vdr krb5kdc[11039](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376261, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T22:44:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T22:44:21Z Disabled control of system clock
WARNING: no policy specified for kudu/127.10.198.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.198.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.198.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790376259650642-11032-0/krb5kdc/kudu_127.10.198.62.keytab.
Entry for principal kudu/127.10.198.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790376259650642-11032-0/krb5kdc/kudu_127.10.198.62.keytab.
I20260925 22:44:21.267086 11032 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrWXm8X/build/debug/bin/kudu
/tmp/dist-test-taskrWXm8X/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790376259650642-11032-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790376259650642-11032-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790376259650642-11032-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790376259650642-11032-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.198.62:33587
--webserver_interface=127.10.198.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790376259650642-11032-0/krb5kdc/kudu_127.10.198.62.keytab
--principal=kudu/127.10.198.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.198.20:45859
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.198.62:33587
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790376259650642-11032-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790376259650642-11032-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790376259650642-11032-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:44:21.386423 11052 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:44:21.386749 11052 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:44:21.386790 11052 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:44:21.390954 11052 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 22:44:21.391041 11052 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:44:21.391062 11052 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:44:21.391078 11052 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 22:44:21.391094 11052 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 22:44:21.397094 11052 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.198.20:45859
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790376259650642-11032-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790376259650642-11032-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.198.62:33587
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790376259650642-11032-0/krb5kdc/kudu_127.10.198.62.keytab
--principal=kudu/127.10.198.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.198.62:33587
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790376259650642-11032-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.198.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11052
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790376259650642-11032-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:09:16 UTC on ccycloud.quasar-afmhyw.root.comops.site
build id 4673379
I20260925 22:44:21.398309 11052 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:44:21.399415 11052 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 22:44:21.409288 11052 server_base.cc:1034] running on GCE node
W20260925 22:44:21.409245 11057 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:44:21.409273 11058 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:44:21.409579 11060 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:44:21.410266 11052 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:44:21.411715 11052 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:44:21.412986 11052 hybrid_clock.cc:648] HybridClock initialized: now 1790376261412944 us; error 64 us; skew 500 ppm
Sep 25 22:44:21 dist-test-slave-5vdr krb5kdc[11039](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376261, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.198.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:44:21.416569 11052 init.cc:377] Logged in from keytab as kudu/127.10.198.62@KRBTEST.COM (short username kudu)
I20260925 22:44:21.418315 11052 webserver.cc:460] Webserver started at http://127.10.198.62:35085/ using document root <none> and password file <none>
I20260925 22:44:21.419275 11052 fs_manager.cc:359] Metadata directory not provided
I20260925 22:44:21.419425 11052 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:44:21.419724 11052 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:44:21.422554 11052 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790376259650642-11032-0/minicluster-data/master-0/data/instance:
uuid: "fafeccc974f2427cad835fa8a76b5af2"
format_stamp: "Formatted at 2026-09-25 22:44:21 on dist-test-slave-5vdr"
server_key: "7e6fc37adac54a98f58fa8fd4f43f0e5"
server_key_iv: "dfdffc094778d65c898cad8df0af9801"
server_key_version: "encryptionkey@0"
I20260925 22:44:21.423331 11052 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790376259650642-11032-0/minicluster-data/master-0/wal/instance:
uuid: "fafeccc974f2427cad835fa8a76b5af2"
format_stamp: "Formatted at 2026-09-25 22:44:21 on dist-test-slave-5vdr"
server_key: "7e6fc37adac54a98f58fa8fd4f43f0e5"
server_key_iv: "dfdffc094778d65c898cad8df0af9801"
server_key_version: "encryptionkey@0"
I20260925 22:44:21.427940 11052 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260925 22:44:21.431092 11067 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:44:21.433030 11052 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 22:44:21.433272 11052 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790376259650642-11032-0/minicluster-data/master-0/data,/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790376259650642-11032-0/minicluster-data/master-0/wal
uuid: "fafeccc974f2427cad835fa8a76b5af2"
format_stamp: "Formatted at 2026-09-25 22:44:21 on dist-test-slave-5vdr"
server_key: "7e6fc37adac54a98f58fa8fd4f43f0e5"
server_key_iv: "dfdffc094778d65c898cad8df0af9801"
server_key_version: "encryptionkey@0"
I20260925 22:44:21.433447 11052 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790376259650642-11032-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790376259650642-11032-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790376259650642-11032-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:44:21.447377 11052 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:44:21.451529 11052 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:44:21.451800 11052 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:44:21.459918 11052 rpc_server.cc:307] RPC server started. Bound to: 127.10.198.62:33587
I20260925 22:44:21.459947 11119 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.198.62:33587 every 8 connection(s)
I20260925 22:44:21.461047 11052 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790376259650642-11032-0/minicluster-data/master-0/data/info.pb
I20260925 22:44:21.464299 11120 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:44:21.464977 11032 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrWXm8X/build/debug/bin/kudu as pid 11052
I20260925 22:44:21.465145 11032 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790376259650642-11032-0/minicluster-data/master-0/wal/instance
I20260925 22:44:21.466758 11032 external_mini_cluster.cc:1428] Setting key 5445e950f0ef60b2dfa582d76569dacf
I20260925 22:44:21.471586 11120 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fafeccc974f2427cad835fa8a76b5af2: Bootstrap starting.
I20260925 22:44:21.474318 11120 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fafeccc974f2427cad835fa8a76b5af2: Neither blocks nor log segments found. Creating new log.
I20260925 22:44:21.475246 11120 log.cc:824] T 00000000000000000000000000000000 P fafeccc974f2427cad835fa8a76b5af2: Log is configured to *not* fsync() on all Append() calls
I20260925 22:44:21.477690 11120 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fafeccc974f2427cad835fa8a76b5af2: No bootstrap required, opened a new log
I20260925 22:44:21.481600 11120 raft_consensus.cc:348] T 00000000000000000000000000000000 P fafeccc974f2427cad835fa8a76b5af2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fafeccc974f2427cad835fa8a76b5af2" member_type: VOTER last_known_addr { host: "127.10.198.62" port: 33587 } }
I20260925 22:44:21.481942 11120 raft_consensus.cc:374] T 00000000000000000000000000000000 P fafeccc974f2427cad835fa8a76b5af2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:44:21.482040 11120 raft_consensus.cc:729] T 00000000000000000000000000000000 P fafeccc974f2427cad835fa8a76b5af2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fafeccc974f2427cad835fa8a76b5af2, State: Initialized, Role: FOLLOWER
I20260925 22:44:21.482559 11120 consensus_queue.cc:260] T 00000000000000000000000000000000 P fafeccc974f2427cad835fa8a76b5af2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fafeccc974f2427cad835fa8a76b5af2" member_type: VOTER last_known_addr { host: "127.10.198.62" port: 33587 } }
I20260925 22:44:21.482736 11120 raft_consensus.cc:388] T 00000000000000000000000000000000 P fafeccc974f2427cad835fa8a76b5af2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 22:44:21.482828 11120 raft_consensus.cc:482] T 00000000000000000000000000000000 P fafeccc974f2427cad835fa8a76b5af2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 22:44:21.482954 11120 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fafeccc974f2427cad835fa8a76b5af2 [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:44:21.484073 11120 raft_consensus.cc:504] T 00000000000000000000000000000000 P fafeccc974f2427cad835fa8a76b5af2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fafeccc974f2427cad835fa8a76b5af2" member_type: VOTER last_known_addr { host: "127.10.198.62" port: 33587 } }
I20260925 22:44:21.484483 11120 leader_election.cc:302] T 00000000000000000000000000000000 P fafeccc974f2427cad835fa8a76b5af2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fafeccc974f2427cad835fa8a76b5af2; no voters:
I20260925 22:44:21.484927 11120 leader_election.cc:288] T 00000000000000000000000000000000 P fafeccc974f2427cad835fa8a76b5af2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 22:44:21.485064 11125 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fafeccc974f2427cad835fa8a76b5af2 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 22:44:21.485342 11125 raft_consensus.cc:686] T 00000000000000000000000000000000 P fafeccc974f2427cad835fa8a76b5af2 [term 1 LEADER]: Becoming Leader. State: Replica: fafeccc974f2427cad835fa8a76b5af2, State: Running, Role: LEADER
I20260925 22:44:21.485939 11125 consensus_queue.cc:237] T 00000000000000000000000000000000 P fafeccc974f2427cad835fa8a76b5af2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fafeccc974f2427cad835fa8a76b5af2" member_type: VOTER last_known_addr { host: "127.10.198.62" port: 33587 } }
I20260925 22:44:21.486528 11120 sys_catalog.cc:564] T 00000000000000000000000000000000 P fafeccc974f2427cad835fa8a76b5af2 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 22:44:21.487739 11125 sys_catalog.cc:455] T 00000000000000000000000000000000 P fafeccc974f2427cad835fa8a76b5af2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fafeccc974f2427cad835fa8a76b5af2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fafeccc974f2427cad835fa8a76b5af2" member_type: VOTER last_known_addr { host: "127.10.198.62" port: 33587 } } }
I20260925 22:44:21.488047 11125 sys_catalog.cc:458] T 00000000000000000000000000000000 P fafeccc974f2427cad835fa8a76b5af2 [sys.catalog]: This master's current role is: LEADER
Sep 25 22:44:21 dist-test-slave-5vdr krb5kdc[11039](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376261, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.198.62@KRBTEST.COM
I20260925 22:44:21.488234 11126 sys_catalog.cc:455] T 00000000000000000000000000000000 P fafeccc974f2427cad835fa8a76b5af2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fafeccc974f2427cad835fa8a76b5af2. Latest consensus state: current_term: 1 leader_uuid: "fafeccc974f2427cad835fa8a76b5af2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fafeccc974f2427cad835fa8a76b5af2" member_type: VOTER last_known_addr { host: "127.10.198.62" port: 33587 } } }
I20260925 22:44:21.488344 11126 sys_catalog.cc:458] T 00000000000000000000000000000000 P fafeccc974f2427cad835fa8a76b5af2 [sys.catalog]: This master's current role is: LEADER
I20260925 22:44:21.488552 11134 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 22:44:21.492529 11134 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 22:44:21.497344 11122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:44:21.470378 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33428 (local address 127.10.198.62:33587)
0925 22:44:21.470901 (+ 523us) server_negotiation.cc:206] Beginning negotiation
0925 22:44:21.470918 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0925 22:44:21.472278 (+ 1360us) server_negotiation.cc:407] Connection header received
0925 22:44:21.476179 (+ 3901us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:44:21.476205 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:44:21.476622 (+ 417us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:44:21.477036 (+ 414us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:44:21.479726 (+ 2690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:44:21.480888 (+ 1162us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:44:21.482468 (+ 1580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:44:21.483157 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:44:21.483370 (+ 213us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:44:21.489641 (+ 6271us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:44:21.489651 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:44:21.489671 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:44:21.489711 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:44:21.492062 (+ 2351us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:44:21.492942 (+ 880us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:44:21.492953 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:44:21.492960 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:44:21.493138 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:44:21.493643 (+ 505us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:44:21.493647 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:44:21.493649 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:44:21.494216 (+ 567us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:44:21.494439 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0925 22:44:21.495534 (+ 1095us) server_negotiation.cc:299] Negotiation successful
0925 22:44:21.496009 (+ 475us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":352,"thread_start_us":193,"threads_started":1}
I20260925 22:44:21.501035 11134 catalog_manager.cc:1347] Generated new cluster ID: 33614f4377864cfeb05246623eabbfe6
I20260925 22:44:21.501124 11134 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 22:44:21.521633 11134 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 22:44:21.523088 11134 catalog_manager.cc:1504] Loading token signing keys...
I20260925 22:44:21.534970 11134 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fafeccc974f2427cad835fa8a76b5af2: Generated new TSK 0
I20260925 22:44:21.536103 11134 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.10.198.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.198.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.198.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790376259650642-11032-0/krb5kdc/kudu_127.10.198.1.keytab.
Entry for principal kudu/127.10.198.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790376259650642-11032-0/krb5kdc/kudu_127.10.198.1.keytab.
I20260925 22:44:21.573097 11032 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrWXm8X/build/debug/bin/kudu
/tmp/dist-test-taskrWXm8X/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790376259650642-11032-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790376259650642-11032-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790376259650642-11032-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790376259650642-11032-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.198.1:0
--local_ip_for_outbound_sockets=127.10.198.1
--webserver_interface=127.10.198.1
--webserver_port=0
--tserver_master_addrs=127.10.198.62:33587
--keytab_file=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790376259650642-11032-0/krb5kdc/kudu_127.10.198.1.keytab
--principal=kudu/127.10.198.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.198.20:45859
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790376259650642-11032-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790376259650642-11032-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790376259650642-11032-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:44:21.702301 11146 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:44:21.702616 11146 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:44:21.702668 11146 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:44:21.707161 11146 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:44:21.707257 11146 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:44:21.707366 11146 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.198.1
I20260925 22:44:21.713275 11146 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.198.20:45859
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790376259650642-11032-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790376259650642-11032-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790376259650642-11032-0/krb5kdc/kudu_127.10.198.1.keytab
--principal=kudu/127.10.198.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.198.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790376259650642-11032-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.198.1
--webserver_port=0
--tserver_master_addrs=127.10.198.62:33587
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11146
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.198.1
--log_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790376259650642-11032-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:09:16 UTC on ccycloud.quasar-afmhyw.root.comops.site
build id 4673379
I20260925 22:44:21.714561 11146 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:44:21.715739 11146 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:44:21.726781 11152 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:44:21.727041 11146 server_base.cc:1034] running on GCE node
W20260925 22:44:21.726781 11151 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:44:21.726781 11154 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:44:21.727658 11146 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:44:21.728569 11146 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:44:21.729847 11146 hybrid_clock.cc:648] HybridClock initialized: now 1790376261729815 us; error 36 us; skew 500 ppm
Sep 25 22:44:21 dist-test-slave-5vdr krb5kdc[11039](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376261, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.198.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:44:21.733381 11146 init.cc:377] Logged in from keytab as kudu/127.10.198.1@KRBTEST.COM (short username kudu)
I20260925 22:44:21.734525 11146 webserver.cc:460] Webserver started at http://127.10.198.1:45693/ using document root <none> and password file <none>
I20260925 22:44:21.735217 11146 fs_manager.cc:359] Metadata directory not provided
I20260925 22:44:21.735335 11146 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:44:21.735615 11146 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:44:21.738107 11146 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790376259650642-11032-0/minicluster-data/ts-0/data/instance:
uuid: "efc8a8649071492292c96e11fd508d43"
format_stamp: "Formatted at 2026-09-25 22:44:21 on dist-test-slave-5vdr"
server_key: "681eb3f933b7389ebbfce57ef1a6cdea"
server_key_iv: "a75c5d300c0005cccd0637afd5ef24a2"
server_key_version: "encryptionkey@0"
I20260925 22:44:21.738809 11146 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790376259650642-11032-0/minicluster-data/ts-0/wal/instance:
uuid: "efc8a8649071492292c96e11fd508d43"
format_stamp: "Formatted at 2026-09-25 22:44:21 on dist-test-slave-5vdr"
server_key: "681eb3f933b7389ebbfce57ef1a6cdea"
server_key_iv: "a75c5d300c0005cccd0637afd5ef24a2"
server_key_version: "encryptionkey@0"
I20260925 22:44:21.743487 11146 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260925 22:44:21.746632 11161 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:44:21.748306 11146 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 22:44:21.748502 11146 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790376259650642-11032-0/minicluster-data/ts-0/data,/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790376259650642-11032-0/minicluster-data/ts-0/wal
uuid: "efc8a8649071492292c96e11fd508d43"
format_stamp: "Formatted at 2026-09-25 22:44:21 on dist-test-slave-5vdr"
server_key: "681eb3f933b7389ebbfce57ef1a6cdea"
server_key_iv: "a75c5d300c0005cccd0637afd5ef24a2"
server_key_version: "encryptionkey@0"
I20260925 22:44:21.748667 11146 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790376259650642-11032-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790376259650642-11032-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790376259650642-11032-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:44:21.771625 11146 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:44:21.775158 11146 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:44:21.775411 11146 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:44:21.776157 11146 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 22:44:21.777244 11146 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:44:21.777348 11146 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:44:21.777457 11146 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:44:21.777510 11146 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 22:44:21.788168 11146 rpc_server.cc:307] RPC server started. Bound to: 127.10.198.1:41591
I20260925 22:44:21.788226 11274 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.198.1:41591 every 8 connection(s)
I20260925 22:44:21.789304 11146 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790376259650642-11032-0/minicluster-data/ts-0/data/info.pb
I20260925 22:44:21.791115 11032 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrWXm8X/build/debug/bin/kudu as pid 11146
I20260925 22:44:21.791220 11032 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790376259650642-11032-0/minicluster-data/ts-0/wal/instance
I20260925 22:44:21.791568 11032 external_mini_cluster.cc:1428] Setting key 423499d3199d12b491d6cf54db8ce7c0
WARNING: no policy specified for kudu/127.10.198.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.198.2@KRBTEST.COM" created.
Sep 25 22:44:21 dist-test-slave-5vdr krb5kdc[11039](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376261, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.198.1@KRBTEST.COM for kudu/127.10.198.62@KRBTEST.COM
I20260925 22:44:21.811421 11122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:44:21.791794 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.198.1:50063 (local address 127.10.198.62:33587)
0925 22:44:21.792010 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0925 22:44:21.792016 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 22:44:21.793543 (+ 1527us) server_negotiation.cc:407] Connection header received
0925 22:44:21.795092 (+ 1549us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:44:21.795095 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:44:21.795151 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:44:21.795332 (+ 181us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:44:21.796901 (+ 1569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:44:21.797818 (+ 917us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:44:21.799253 (+ 1435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:44:21.799731 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:44:21.799905 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:44:21.806277 (+ 6372us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:44:21.806281 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:44:21.806284 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:44:21.806317 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:44:21.808401 (+ 2084us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:44:21.809174 (+ 773us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:44:21.809181 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:44:21.809186 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:44:21.809307 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:44:21.809816 (+ 509us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:44:21.809821 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:44:21.809822 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:44:21.810121 (+ 299us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:44:21.810265 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0925 22:44:21.810957 (+ 692us) server_negotiation.cc:299] Negotiation successful
0925 22:44:21.811161 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":98}
I20260925 22:44:21.812481 11277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:44:21.792410 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.198.62:33587 (local address 127.10.198.1:50063)
0925 22:44:21.793370 (+ 960us) negotiation.cc:107] Waiting for socket to connect
0925 22:44:21.793448 (+ 78us) client_negotiation.cc:174] Beginning negotiation
0925 22:44:21.794760 (+ 1312us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:44:21.795585 (+ 825us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:44:21.795594 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:44:21.796117 (+ 523us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:44:21.796702 (+ 585us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:44:21.796718 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:44:21.798013 (+ 1295us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:44:21.798020 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:44:21.799071 (+ 1051us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:44:21.799083 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:44:21.800011 (+ 928us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:44:21.800014 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:44:21.800219 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:44:21.801079 (+ 860us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:44:21.801099 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:44:21.805966 (+ 4867us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:44:21.808598 (+ 2632us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:44:21.808606 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:44:21.808625 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:44:21.809005 (+ 380us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:44:21.809434 (+ 429us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:44:21.809438 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:44:21.809440 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:44:21.809657 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:44:21.810284 (+ 627us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:44:21.810290 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:44:21.810616 (+ 326us) client_negotiation.cc:769] Sending connection context
0925 22:44:21.810820 (+ 204us) client_negotiation.cc:240] Negotiation successful
0925 22:44:21.811142 (+ 322us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":674,"thread_start_us":195,"threads_started":1}
I20260925 22:44:21.814349 11275 heartbeater.cc:347] Connected to a master server at 127.10.198.62:33587
I20260925 22:44:21.814688 11275 heartbeater.cc:464] Registering TS with master...
I20260925 22:44:21.815366 11275 heartbeater.cc:511] Master 127.10.198.62:33587 requested a full tablet report, sending...
I20260925 22:44:21.817476 11084 ts_manager.cc:194] Registered new tserver with Master: efc8a8649071492292c96e11fd508d43 (127.10.198.1:41591)
I20260925 22:44:21.818894 11084 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.198.1@KRBTEST.COM'} at 127.10.198.1:50063
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.198.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790376259650642-11032-0/krb5kdc/kudu_127.10.198.2.keytab.
Entry for principal kudu/127.10.198.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790376259650642-11032-0/krb5kdc/kudu_127.10.198.2.keytab.
I20260925 22:44:21.823449 11032 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrWXm8X/build/debug/bin/kudu
/tmp/dist-test-taskrWXm8X/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790376259650642-11032-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790376259650642-11032-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790376259650642-11032-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790376259650642-11032-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.198.2:0
--local_ip_for_outbound_sockets=127.10.198.2
--webserver_interface=127.10.198.2
--webserver_port=0
--tserver_master_addrs=127.10.198.62:33587
--keytab_file=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790376259650642-11032-0/krb5kdc/kudu_127.10.198.2.keytab
--principal=kudu/127.10.198.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.198.20:45859
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790376259650642-11032-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790376259650642-11032-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790376259650642-11032-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:44:21.947233 11280 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:44:21.947554 11280 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:44:21.947603 11280 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:44:21.952251 11280 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:44:21.952332 11280 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:44:21.952443 11280 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.198.2
I20260925 22:44:21.958417 11280 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.198.20:45859
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790376259650642-11032-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790376259650642-11032-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790376259650642-11032-0/krb5kdc/kudu_127.10.198.2.keytab
--principal=kudu/127.10.198.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.198.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790376259650642-11032-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.198.2
--webserver_port=0
--tserver_master_addrs=127.10.198.62:33587
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11280
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.198.2
--log_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790376259650642-11032-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:09:16 UTC on ccycloud.quasar-afmhyw.root.comops.site
build id 4673379
I20260925 22:44:21.959661 11280 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:44:21.960870 11280 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:44:21.971990 11288 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:44:21.971994 11286 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:44:21.971994 11285 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:44:21.972354 11280 server_base.cc:1034] running on GCE node
I20260925 22:44:21.972882 11280 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:44:21.973647 11280 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:44:21.974864 11280 hybrid_clock.cc:648] HybridClock initialized: now 1790376261974851 us; error 39 us; skew 500 ppm
Sep 25 22:44:21 dist-test-slave-5vdr krb5kdc[11039](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376261, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.198.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:44:21.978076 11280 init.cc:377] Logged in from keytab as kudu/127.10.198.2@KRBTEST.COM (short username kudu)
I20260925 22:44:21.979223 11280 webserver.cc:460] Webserver started at http://127.10.198.2:44437/ using document root <none> and password file <none>
I20260925 22:44:21.979952 11280 fs_manager.cc:359] Metadata directory not provided
I20260925 22:44:21.980054 11280 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:44:21.980304 11280 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:44:21.982663 11280 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790376259650642-11032-0/minicluster-data/ts-1/data/instance:
uuid: "1f9838234b8e42bba3023d687480aebc"
format_stamp: "Formatted at 2026-09-25 22:44:21 on dist-test-slave-5vdr"
server_key: "c2049efec61be492fb69bcd81072e562"
server_key_iv: "16fce6b894e333990d185958457ae137"
server_key_version: "encryptionkey@0"
I20260925 22:44:21.983389 11280 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790376259650642-11032-0/minicluster-data/ts-1/wal/instance:
uuid: "1f9838234b8e42bba3023d687480aebc"
format_stamp: "Formatted at 2026-09-25 22:44:21 on dist-test-slave-5vdr"
server_key: "c2049efec61be492fb69bcd81072e562"
server_key_iv: "16fce6b894e333990d185958457ae137"
server_key_version: "encryptionkey@0"
I20260925 22:44:21.988142 11280 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.008s sys 0.000s
I20260925 22:44:21.991328 11295 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:44:21.993111 11280 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 22:44:21.993335 11280 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790376259650642-11032-0/minicluster-data/ts-1/data,/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790376259650642-11032-0/minicluster-data/ts-1/wal
uuid: "1f9838234b8e42bba3023d687480aebc"
format_stamp: "Formatted at 2026-09-25 22:44:21 on dist-test-slave-5vdr"
server_key: "c2049efec61be492fb69bcd81072e562"
server_key_iv: "16fce6b894e333990d185958457ae137"
server_key_version: "encryptionkey@0"
I20260925 22:44:21.993495 11280 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790376259650642-11032-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790376259650642-11032-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790376259650642-11032-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:44:22.014709 11280 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:44:22.018782 11280 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:44:22.019043 11280 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:44:22.019771 11280 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 22:44:22.020843 11280 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:44:22.020921 11280 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:44:22.020998 11280 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:44:22.021040 11280 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 22:44:22.030648 11280 rpc_server.cc:307] RPC server started. Bound to: 127.10.198.2:39497
I20260925 22:44:22.030824 11408 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.198.2:39497 every 8 connection(s)
I20260925 22:44:22.031737 11280 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790376259650642-11032-0/minicluster-data/ts-1/data/info.pb
I20260925 22:44:22.041080 11032 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrWXm8X/build/debug/bin/kudu as pid 11280
I20260925 22:44:22.041235 11032 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790376259650642-11032-0/minicluster-data/ts-1/wal/instance
I20260925 22:44:22.041517 11032 external_mini_cluster.cc:1428] Setting key e82eb4d4ec31ceb8d14396f23a58cf48
Sep 25 22:44:22 dist-test-slave-5vdr krb5kdc[11039](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376261, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.198.2@KRBTEST.COM for kudu/127.10.198.62@KRBTEST.COM
I20260925 22:44:22.049455 11122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:44:22.034018 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.198.2:50753 (local address 127.10.198.62:33587)
0925 22:44:22.034201 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0925 22:44:22.034206 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:44:22.035594 (+ 1388us) server_negotiation.cc:407] Connection header received
0925 22:44:22.036643 (+ 1049us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:44:22.036646 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:44:22.036702 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:44:22.036796 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:44:22.038517 (+ 1721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:44:22.039051 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:44:22.040800 (+ 1749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:44:22.041045 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:44:22.041148 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:44:22.044947 (+ 3799us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:44:22.044951 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:44:22.044953 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:44:22.044983 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:44:22.046600 (+ 1617us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:44:22.047285 (+ 685us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:44:22.047288 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:44:22.047289 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:44:22.047379 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:44:22.047811 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:44:22.047817 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:44:22.047824 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:44:22.048105 (+ 281us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:44:22.048226 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0925 22:44:22.049081 (+ 855us) server_negotiation.cc:299] Negotiation successful
0925 22:44:22.049265 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260925 22:44:22.050741 11411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:44:22.034487 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.198.62:33587 (local address 127.10.198.2:50753)
0925 22:44:22.035413 (+ 926us) negotiation.cc:107] Waiting for socket to connect
0925 22:44:22.035469 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0925 22:44:22.036364 (+ 895us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:44:22.037058 (+ 694us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:44:22.037067 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:44:22.037565 (+ 498us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:44:22.038331 (+ 766us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:44:22.038351 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:44:22.039218 (+ 867us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:44:22.039221 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:44:22.040613 (+ 1392us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:44:22.040628 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:44:22.041315 (+ 687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:44:22.041322 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:44:22.041588 (+ 266us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:44:22.042446 (+ 858us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:44:22.042465 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:44:22.044712 (+ 2247us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:44:22.046771 (+ 2059us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:44:22.046779 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:44:22.046803 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:44:22.047140 (+ 337us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:44:22.047489 (+ 349us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:44:22.047492 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:44:22.047495 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:44:22.047689 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:44:22.048227 (+ 538us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:44:22.048238 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:44:22.048551 (+ 313us) client_negotiation.cc:769] Sending connection context
0925 22:44:22.048896 (+ 345us) client_negotiation.cc:240] Negotiation successful
0925 22:44:22.049403 (+ 507us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":654,"thread_start_us":168,"threads_started":1}
WARNING: no policy specified for kudu/127.10.198.3@KRBTEST.COM; defaulting to no policy
I20260925 22:44:22.052466 11409 heartbeater.cc:347] Connected to a master server at 127.10.198.62:33587
I20260925 22:44:22.052855 11409 heartbeater.cc:464] Registering TS with master...
I20260925 22:44:22.053599 11409 heartbeater.cc:511] Master 127.10.198.62:33587 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.198.3@KRBTEST.COM" created.
I20260925 22:44:22.055169 11084 ts_manager.cc:194] Registered new tserver with Master: 1f9838234b8e42bba3023d687480aebc (127.10.198.2:39497)
I20260925 22:44:22.055833 11084 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.198.2@KRBTEST.COM'} at 127.10.198.2:50753
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.198.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790376259650642-11032-0/krb5kdc/kudu_127.10.198.3.keytab.
Entry for principal kudu/127.10.198.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790376259650642-11032-0/krb5kdc/kudu_127.10.198.3.keytab.
I20260925 22:44:22.071478 11032 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrWXm8X/build/debug/bin/kudu
/tmp/dist-test-taskrWXm8X/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790376259650642-11032-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790376259650642-11032-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790376259650642-11032-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790376259650642-11032-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.198.3:0
--local_ip_for_outbound_sockets=127.10.198.3
--webserver_interface=127.10.198.3
--webserver_port=0
--tserver_master_addrs=127.10.198.62:33587
--keytab_file=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790376259650642-11032-0/krb5kdc/kudu_127.10.198.3.keytab
--principal=kudu/127.10.198.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.198.20:45859
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790376259650642-11032-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790376259650642-11032-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790376259650642-11032-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:44:22.197950 11414 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:44:22.198279 11414 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:44:22.198381 11414 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:44:22.203706 11414 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:44:22.203944 11414 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:44:22.204079 11414 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.198.3
I20260925 22:44:22.212011 11414 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.198.20:45859
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790376259650642-11032-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790376259650642-11032-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790376259650642-11032-0/krb5kdc/kudu_127.10.198.3.keytab
--principal=kudu/127.10.198.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.198.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790376259650642-11032-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.198.3
--webserver_port=0
--tserver_master_addrs=127.10.198.62:33587
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11414
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.198.3
--log_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790376259650642-11032-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:09:16 UTC on ccycloud.quasar-afmhyw.root.comops.site
build id 4673379
I20260925 22:44:22.213310 11414 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:44:22.214514 11414 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:44:22.226959 11419 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:44:22.226989 11420 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:44:22.226959 11422 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:44:22.227269 11414 server_base.cc:1034] running on GCE node
I20260925 22:44:22.227829 11414 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:44:22.228726 11414 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:44:22.229946 11414 hybrid_clock.cc:648] HybridClock initialized: now 1790376262229911 us; error 54 us; skew 500 ppm
Sep 25 22:44:22 dist-test-slave-5vdr krb5kdc[11039](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376262, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.198.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:44:22.233248 11414 init.cc:377] Logged in from keytab as kudu/127.10.198.3@KRBTEST.COM (short username kudu)
I20260925 22:44:22.234359 11414 webserver.cc:460] Webserver started at http://127.10.198.3:37891/ using document root <none> and password file <none>
I20260925 22:44:22.235024 11414 fs_manager.cc:359] Metadata directory not provided
I20260925 22:44:22.235132 11414 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:44:22.235397 11414 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:44:22.237862 11414 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790376259650642-11032-0/minicluster-data/ts-2/data/instance:
uuid: "2c23e67fbef04e64802a514bfb53711e"
format_stamp: "Formatted at 2026-09-25 22:44:22 on dist-test-slave-5vdr"
server_key: "2959b79e6e6bbdf1c876d46c940ddf7a"
server_key_iv: "4e501504ed36a7d9a11e50f13a92fcf5"
server_key_version: "encryptionkey@0"
I20260925 22:44:22.238572 11414 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790376259650642-11032-0/minicluster-data/ts-2/wal/instance:
uuid: "2c23e67fbef04e64802a514bfb53711e"
format_stamp: "Formatted at 2026-09-25 22:44:22 on dist-test-slave-5vdr"
server_key: "2959b79e6e6bbdf1c876d46c940ddf7a"
server_key_iv: "4e501504ed36a7d9a11e50f13a92fcf5"
server_key_version: "encryptionkey@0"
I20260925 22:44:22.242992 11414 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.002s
I20260925 22:44:22.246008 11429 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:44:22.247486 11414 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260925 22:44:22.247659 11414 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790376259650642-11032-0/minicluster-data/ts-2/data,/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790376259650642-11032-0/minicluster-data/ts-2/wal
uuid: "2c23e67fbef04e64802a514bfb53711e"
format_stamp: "Formatted at 2026-09-25 22:44:22 on dist-test-slave-5vdr"
server_key: "2959b79e6e6bbdf1c876d46c940ddf7a"
server_key_iv: "4e501504ed36a7d9a11e50f13a92fcf5"
server_key_version: "encryptionkey@0"
I20260925 22:44:22.247808 11414 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790376259650642-11032-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790376259650642-11032-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790376259650642-11032-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:44:22.267633 11414 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:44:22.271019 11414 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:44:22.271270 11414 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:44:22.271979 11414 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 22:44:22.272872 11414 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:44:22.272950 11414 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:44:22.273075 11414 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:44:22.273115 11414 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 22:44:22.282759 11414 rpc_server.cc:307] RPC server started. Bound to: 127.10.198.3:36059
I20260925 22:44:22.282791 11542 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.198.3:36059 every 8 connection(s)
I20260925 22:44:22.283797 11414 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790376259650642-11032-0/minicluster-data/ts-2/data/info.pb
I20260925 22:44:22.289568 11032 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrWXm8X/build/debug/bin/kudu as pid 11414
I20260925 22:44:22.289687 11032 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790376259650642-11032-0/minicluster-data/ts-2/wal/instance
I20260925 22:44:22.289978 11032 external_mini_cluster.cc:1428] Setting key 03739db4444197dbe25cfe46be27f550
Sep 25 22:44:22 dist-test-slave-5vdr krb5kdc[11039](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376262, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.198.3@KRBTEST.COM for kudu/127.10.198.62@KRBTEST.COM
I20260925 22:44:22.299871 11122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:44:22.285889 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.198.3:49121 (local address 127.10.198.62:33587)
0925 22:44:22.286073 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0925 22:44:22.286078 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:44:22.287383 (+ 1305us) server_negotiation.cc:407] Connection header received
0925 22:44:22.288527 (+ 1144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:44:22.288530 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:44:22.288588 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:44:22.288680 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:44:22.290233 (+ 1553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:44:22.290957 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:44:22.292617 (+ 1660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:44:22.292949 (+ 332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:44:22.293038 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:44:22.295761 (+ 2723us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:44:22.295764 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:44:22.295766 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:44:22.295801 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:44:22.297303 (+ 1502us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:44:22.298026 (+ 723us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:44:22.298029 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:44:22.298031 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:44:22.298130 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:44:22.298523 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:44:22.298526 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:44:22.298528 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:44:22.298802 (+ 274us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:44:22.298910 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0925 22:44:22.299508 (+ 598us) server_negotiation.cc:299] Negotiation successful
0925 22:44:22.299659 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260925 22:44:22.300932 11545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:44:22.286365 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.198.62:33587 (local address 127.10.198.3:49121)
0925 22:44:22.287154 (+ 789us) negotiation.cc:107] Waiting for socket to connect
0925 22:44:22.287234 (+ 80us) client_negotiation.cc:174] Beginning negotiation
0925 22:44:22.288276 (+ 1042us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:44:22.288954 (+ 678us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:44:22.288962 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:44:22.289389 (+ 427us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:44:22.290007 (+ 618us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:44:22.290047 (+ 40us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:44:22.291113 (+ 1066us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:44:22.291116 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:44:22.292476 (+ 1360us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:44:22.292489 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:44:22.293124 (+ 635us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:44:22.293128 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:44:22.293299 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:44:22.293921 (+ 622us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:44:22.293939 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:44:22.295536 (+ 1597us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:44:22.297470 (+ 1934us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:44:22.297477 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:44:22.297504 (+ 27us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:44:22.297903 (+ 399us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:44:22.298236 (+ 333us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:44:22.298239 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:44:22.298242 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:44:22.298414 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:44:22.298924 (+ 510us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:44:22.298934 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:44:22.299235 (+ 301us) client_negotiation.cc:769] Sending connection context
0925 22:44:22.299464 (+ 229us) client_negotiation.cc:240] Negotiation successful
0925 22:44:22.299847 (+ 383us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":565,"thread_start_us":166,"threads_started":1}
I20260925 22:44:22.302398 11543 heartbeater.cc:347] Connected to a master server at 127.10.198.62:33587
I20260925 22:44:22.302719 11543 heartbeater.cc:464] Registering TS with master...
I20260925 22:44:22.303335 11543 heartbeater.cc:511] Master 127.10.198.62:33587 requested a full tablet report, sending...
I20260925 22:44:22.304800 11084 ts_manager.cc:194] Registered new tserver with Master: 2c23e67fbef04e64802a514bfb53711e (127.10.198.3:36059)
I20260925 22:44:22.305552 11084 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.198.3@KRBTEST.COM'} at 127.10.198.3:49121
I20260925 22:44:22.305747 11032 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 25 22:44:22 dist-test-slave-5vdr krb5kdc[11039](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376262, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 25 22:44:22 dist-test-slave-5vdr krb5kdc[11039](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376262, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.10.198.1@KRBTEST.COM
I20260925 22:44:22.334283 11550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:44:22.323715 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33216 (local address 127.10.198.1:41591)
0925 22:44:22.324061 (+ 346us) server_negotiation.cc:206] Beginning negotiation
0925 22:44:22.324067 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 22:44:22.324239 (+ 172us) server_negotiation.cc:407] Connection header received
0925 22:44:22.324302 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:44:22.324308 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:44:22.324509 (+ 201us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:44:22.324670 (+ 161us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:44:22.325429 (+ 759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:44:22.326225 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:44:22.327106 (+ 881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:44:22.327551 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:44:22.327646 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:44:22.330318 (+ 2672us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:44:22.330330 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:44:22.330345 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:44:22.330376 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:44:22.332248 (+ 1872us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:44:22.332763 (+ 515us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:44:22.332769 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:44:22.332770 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:44:22.332875 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:44:22.333216 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:44:22.333219 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:44:22.333220 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:44:22.333489 (+ 269us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:44:22.333658 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0925 22:44:22.333868 (+ 210us) server_negotiation.cc:299] Negotiation successful
0925 22:44:22.334048 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":232,"thread_start_us":134,"threads_started":1}
W20260925 22:44:22.335470 11189 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33216
Sep 25 22:44:22 dist-test-slave-5vdr krb5kdc[11039](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376262, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 25 22:44:22 dist-test-slave-5vdr krb5kdc[11039](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376262, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.198.1@KRBTEST.COM
I20260925 22:44:22.365671 11550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:44:22.355998 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33230 (local address 127.10.198.1:41591)
0925 22:44:22.356158 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0925 22:44:22.356162 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:44:22.356479 (+ 317us) server_negotiation.cc:407] Connection header received
0925 22:44:22.356669 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:44:22.356672 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:44:22.356732 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:44:22.356820 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:44:22.357582 (+ 762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:44:22.358151 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:44:22.359365 (+ 1214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:44:22.359685 (+ 320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:44:22.359817 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:44:22.362423 (+ 2606us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:44:22.362427 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:44:22.362430 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:44:22.362472 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:44:22.363802 (+ 1330us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:44:22.364327 (+ 525us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:44:22.364331 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:44:22.364333 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:44:22.364432 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:44:22.364752 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:44:22.364756 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:44:22.364758 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:44:22.364953 (+ 195us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:44:22.365113 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0925 22:44:22.365368 (+ 255us) server_negotiation.cc:299] Negotiation successful
0925 22:44:22.365505 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260925 22:44:22.366873 11032 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrWXm8X/build/debug/bin/kudu with pid 11146
I20260925 22:44:22.373095 11032 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrWXm8X/build/debug/bin/kudu with pid 11280
I20260925 22:44:22.378978 11032 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrWXm8X/build/debug/bin/kudu with pid 11414
I20260925 22:44:22.384845 11032 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrWXm8X/build/debug/bin/kudu with pid 11052
2026-09-25T22:44:22Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (2724 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790376259650642-11032-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 22:44:22 dist-test-slave-5vdr krb5kdc[11557](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 22:44:22 dist-test-slave-5vdr krb5kdc[11557](info): set up 2 sockets
Sep 25 22:44:22 dist-test-slave-5vdr krb5kdc[11557](info): commencing operation
krb5kdc: starting...
W20260925 22:44:23.427635 11032 mini_kdc.cc:121] Time spent starting KDC: real 1.019s user 0.004s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790376259650642-11032-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790376259650642-11032-0/krb5kdc/test-user.keytab.
Sep 25 22:44:23 dist-test-slave-5vdr krb5kdc[11557](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376263, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T22:44:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T22:44:23Z Disabled control of system clock
WARNING: no policy specified for kudu/127.10.198.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.198.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.198.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790376259650642-11032-0/krb5kdc/kudu_127.10.198.62.keytab.
Entry for principal kudu/127.10.198.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790376259650642-11032-0/krb5kdc/kudu_127.10.198.62.keytab.
I20260925 22:44:23.575268 11032 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrWXm8X/build/debug/bin/kudu
/tmp/dist-test-taskrWXm8X/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790376259650642-11032-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790376259650642-11032-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790376259650642-11032-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790376259650642-11032-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.198.62:41253
--webserver_interface=127.10.198.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790376259650642-11032-0/krb5kdc/kudu_127.10.198.62.keytab
--principal=kudu/127.10.198.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.198.20:36283
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.198.62:41253
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790376259650642-11032-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790376259650642-11032-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790376259650642-11032-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:44:23.697364 11570 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:44:23.697688 11570 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:44:23.697733 11570 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:44:23.702302 11570 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 22:44:23.702380 11570 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:44:23.702400 11570 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:44:23.702418 11570 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 22:44:23.702435 11570 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 22:44:23.708549 11570 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.198.20:36283
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790376259650642-11032-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790376259650642-11032-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.198.62:41253
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790376259650642-11032-0/krb5kdc/kudu_127.10.198.62.keytab
--principal=kudu/127.10.198.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.198.62:41253
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790376259650642-11032-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.198.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11570
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790376259650642-11032-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:09:16 UTC on ccycloud.quasar-afmhyw.root.comops.site
build id 4673379
I20260925 22:44:23.709817 11570 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:44:23.710985 11570 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:44:23.720955 11578 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:44:23.720969 11576 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:44:23.720984 11575 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:44:23.721174 11570 server_base.cc:1034] running on GCE node
I20260925 22:44:23.722112 11570 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:44:23.723466 11570 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:44:23.724690 11570 hybrid_clock.cc:648] HybridClock initialized: now 1790376263724648 us; error 66 us; skew 500 ppm
Sep 25 22:44:23 dist-test-slave-5vdr krb5kdc[11557](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376263, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.198.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:44:23.727983 11570 init.cc:377] Logged in from keytab as kudu/127.10.198.62@KRBTEST.COM (short username kudu)
I20260925 22:44:23.729203 11570 webserver.cc:460] Webserver started at http://127.10.198.62:43419/ using document root <none> and password file <none>
I20260925 22:44:23.729876 11570 fs_manager.cc:359] Metadata directory not provided
I20260925 22:44:23.729990 11570 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:44:23.730253 11570 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:44:23.732813 11570 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790376259650642-11032-0/minicluster-data/master-0/data/instance:
uuid: "78fe74f4da614a488e4235f319b88ef6"
format_stamp: "Formatted at 2026-09-25 22:44:23 on dist-test-slave-5vdr"
server_key: "e83e523dfc881ddcff9d79a4bb22276e"
server_key_iv: "42b4ff0f8c5b93511bbb660a5a50fe81"
server_key_version: "encryptionkey@0"
I20260925 22:44:23.733501 11570 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790376259650642-11032-0/minicluster-data/master-0/wal/instance:
uuid: "78fe74f4da614a488e4235f319b88ef6"
format_stamp: "Formatted at 2026-09-25 22:44:23 on dist-test-slave-5vdr"
server_key: "e83e523dfc881ddcff9d79a4bb22276e"
server_key_iv: "42b4ff0f8c5b93511bbb660a5a50fe81"
server_key_version: "encryptionkey@0"
I20260925 22:44:23.737954 11570 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260925 22:44:23.740940 11585 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:44:23.742486 11570 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 22:44:23.742681 11570 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790376259650642-11032-0/minicluster-data/master-0/data,/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790376259650642-11032-0/minicluster-data/master-0/wal
uuid: "78fe74f4da614a488e4235f319b88ef6"
format_stamp: "Formatted at 2026-09-25 22:44:23 on dist-test-slave-5vdr"
server_key: "e83e523dfc881ddcff9d79a4bb22276e"
server_key_iv: "42b4ff0f8c5b93511bbb660a5a50fe81"
server_key_version: "encryptionkey@0"
I20260925 22:44:23.742841 11570 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790376259650642-11032-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790376259650642-11032-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790376259650642-11032-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:44:23.760620 11570 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:44:23.764375 11570 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:44:23.764662 11570 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:44:23.773092 11570 rpc_server.cc:307] RPC server started. Bound to: 127.10.198.62:41253
I20260925 22:44:23.773121 11637 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.198.62:41253 every 8 connection(s)
I20260925 22:44:23.774223 11570 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790376259650642-11032-0/minicluster-data/master-0/data/info.pb
I20260925 22:44:23.777554 11638 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:44:23.783207 11032 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrWXm8X/build/debug/bin/kudu as pid 11570
I20260925 22:44:23.783342 11032 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790376259650642-11032-0/minicluster-data/master-0/wal/instance
I20260925 22:44:23.783609 11032 external_mini_cluster.cc:1428] Setting key c2147817d6a237f6d5b7538e91080d44
I20260925 22:44:23.784324 11638 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 78fe74f4da614a488e4235f319b88ef6: Bootstrap starting.
I20260925 22:44:23.787050 11638 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 78fe74f4da614a488e4235f319b88ef6: Neither blocks nor log segments found. Creating new log.
I20260925 22:44:23.787952 11638 log.cc:824] T 00000000000000000000000000000000 P 78fe74f4da614a488e4235f319b88ef6: Log is configured to *not* fsync() on all Append() calls
I20260925 22:44:23.790066 11638 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 78fe74f4da614a488e4235f319b88ef6: No bootstrap required, opened a new log
Sep 25 22:44:23 dist-test-slave-5vdr krb5kdc[11557](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376263, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.198.62@KRBTEST.COM
I20260925 22:44:23.794054 11638 raft_consensus.cc:348] T 00000000000000000000000000000000 P 78fe74f4da614a488e4235f319b88ef6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "78fe74f4da614a488e4235f319b88ef6" member_type: VOTER last_known_addr { host: "127.10.198.62" port: 41253 } }
I20260925 22:44:23.794442 11638 raft_consensus.cc:374] T 00000000000000000000000000000000 P 78fe74f4da614a488e4235f319b88ef6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:44:23.794557 11638 raft_consensus.cc:729] T 00000000000000000000000000000000 P 78fe74f4da614a488e4235f319b88ef6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 78fe74f4da614a488e4235f319b88ef6, State: Initialized, Role: FOLLOWER
I20260925 22:44:23.795176 11638 consensus_queue.cc:260] T 00000000000000000000000000000000 P 78fe74f4da614a488e4235f319b88ef6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "78fe74f4da614a488e4235f319b88ef6" member_type: VOTER last_known_addr { host: "127.10.198.62" port: 41253 } }
I20260925 22:44:23.795358 11638 raft_consensus.cc:388] T 00000000000000000000000000000000 P 78fe74f4da614a488e4235f319b88ef6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 22:44:23.795457 11638 raft_consensus.cc:482] T 00000000000000000000000000000000 P 78fe74f4da614a488e4235f319b88ef6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 22:44:23.795590 11638 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 78fe74f4da614a488e4235f319b88ef6 [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:44:23.796692 11638 raft_consensus.cc:504] T 00000000000000000000000000000000 P 78fe74f4da614a488e4235f319b88ef6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "78fe74f4da614a488e4235f319b88ef6" member_type: VOTER last_known_addr { host: "127.10.198.62" port: 41253 } }
I20260925 22:44:23.797108 11638 leader_election.cc:302] T 00000000000000000000000000000000 P 78fe74f4da614a488e4235f319b88ef6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 78fe74f4da614a488e4235f319b88ef6; no voters:
I20260925 22:44:23.797478 11638 leader_election.cc:288] T 00000000000000000000000000000000 P 78fe74f4da614a488e4235f319b88ef6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 22:44:23.797583 11643 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 78fe74f4da614a488e4235f319b88ef6 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 22:44:23.797803 11643 raft_consensus.cc:686] T 00000000000000000000000000000000 P 78fe74f4da614a488e4235f319b88ef6 [term 1 LEADER]: Becoming Leader. State: Replica: 78fe74f4da614a488e4235f319b88ef6, State: Running, Role: LEADER
I20260925 22:44:23.798277 11643 consensus_queue.cc:237] T 00000000000000000000000000000000 P 78fe74f4da614a488e4235f319b88ef6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "78fe74f4da614a488e4235f319b88ef6" member_type: VOTER last_known_addr { host: "127.10.198.62" port: 41253 } }
I20260925 22:44:23.799034 11638 sys_catalog.cc:564] T 00000000000000000000000000000000 P 78fe74f4da614a488e4235f319b88ef6 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 22:44:23.799928 11645 sys_catalog.cc:455] T 00000000000000000000000000000000 P 78fe74f4da614a488e4235f319b88ef6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 78fe74f4da614a488e4235f319b88ef6. Latest consensus state: current_term: 1 leader_uuid: "78fe74f4da614a488e4235f319b88ef6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "78fe74f4da614a488e4235f319b88ef6" member_type: VOTER last_known_addr { host: "127.10.198.62" port: 41253 } } }
I20260925 22:44:23.800086 11645 sys_catalog.cc:458] T 00000000000000000000000000000000 P 78fe74f4da614a488e4235f319b88ef6 [sys.catalog]: This master's current role is: LEADER
I20260925 22:44:23.800514 11644 sys_catalog.cc:455] T 00000000000000000000000000000000 P 78fe74f4da614a488e4235f319b88ef6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "78fe74f4da614a488e4235f319b88ef6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "78fe74f4da614a488e4235f319b88ef6" member_type: VOTER last_known_addr { host: "127.10.198.62" port: 41253 } } }
I20260925 22:44:23.800695 11644 sys_catalog.cc:458] T 00000000000000000000000000000000 P 78fe74f4da614a488e4235f319b88ef6 [sys.catalog]: This master's current role is: LEADER
I20260925 22:44:23.801832 11641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:44:23.785219 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39020 (local address 127.10.198.62:41253)
0925 22:44:23.785737 (+ 518us) server_negotiation.cc:206] Beginning negotiation
0925 22:44:23.785754 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0925 22:44:23.785795 (+ 41us) server_negotiation.cc:407] Connection header received
0925 22:44:23.786562 (+ 767us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:44:23.786582 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:44:23.786983 (+ 401us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:44:23.787387 (+ 404us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:44:23.788355 (+ 968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:44:23.789391 (+ 1036us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:44:23.790347 (+ 956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:44:23.790872 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:44:23.791067 (+ 195us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:44:23.793788 (+ 2721us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:44:23.793794 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:44:23.793812 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:44:23.793848 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:44:23.796176 (+ 2328us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:44:23.796736 (+ 560us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:44:23.796743 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:44:23.796751 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:44:23.796949 (+ 198us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:44:23.797278 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:44:23.797283 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:44:23.797285 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:44:23.799146 (+ 1861us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:44:23.799398 (+ 252us) server_negotiation.cc:1035] Waiting for connection context
0925 22:44:23.800238 (+ 840us) server_negotiation.cc:299] Negotiation successful
0925 22:44:23.800636 (+ 398us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":359,"thread_start_us":159,"threads_started":1}
W20260925 22:44:23.804546 11658 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 78fe74f4da614a488e4235f319b88ef6: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 22:44:23.804675 11658 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 22:44:23.804742 11652 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 22:44:23.805718 11652 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 22:44:23.812572 11652 catalog_manager.cc:1347] Generated new cluster ID: f5923ba991cc468b8bfc2d749407f745
I20260925 22:44:23.812673 11652 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 22:44:23.841017 11652 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 22:44:23.842447 11652 catalog_manager.cc:1504] Loading token signing keys...
I20260925 22:44:23.851187 11652 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 78fe74f4da614a488e4235f319b88ef6: Generated new TSK 0
I20260925 22:44:23.852253 11652 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.10.198.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.198.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.198.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790376259650642-11032-0/krb5kdc/kudu_127.10.198.1.keytab.
Entry for principal kudu/127.10.198.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790376259650642-11032-0/krb5kdc/kudu_127.10.198.1.keytab.
I20260925 22:44:23.884123 11032 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrWXm8X/build/debug/bin/kudu
/tmp/dist-test-taskrWXm8X/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790376259650642-11032-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790376259650642-11032-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790376259650642-11032-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790376259650642-11032-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.198.1:0
--local_ip_for_outbound_sockets=127.10.198.1
--webserver_interface=127.10.198.1
--webserver_port=0
--tserver_master_addrs=127.10.198.62:41253
--keytab_file=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790376259650642-11032-0/krb5kdc/kudu_127.10.198.1.keytab
--principal=kudu/127.10.198.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.198.20:36283
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790376259650642-11032-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790376259650642-11032-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790376259650642-11032-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:44:24.009475 11664 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:44:24.009814 11664 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:44:24.009863 11664 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:44:24.014379 11664 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:44:24.014477 11664 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:44:24.014541 11664 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.198.1
I20260925 22:44:24.020846 11664 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.198.20:36283
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790376259650642-11032-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790376259650642-11032-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790376259650642-11032-0/krb5kdc/kudu_127.10.198.1.keytab
--principal=kudu/127.10.198.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.198.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790376259650642-11032-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.198.1
--webserver_port=0
--tserver_master_addrs=127.10.198.62:41253
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11664
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.198.1
--log_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790376259650642-11032-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:09:16 UTC on ccycloud.quasar-afmhyw.root.comops.site
build id 4673379
I20260925 22:44:24.022082 11664 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:44:24.023268 11664 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 22:44:24.034761 11664 server_base.cc:1034] running on GCE node
W20260925 22:44:24.034715 11672 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:44:24.034708 11670 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:44:24.034708 11669 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:44:24.035499 11664 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:44:24.036381 11664 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:44:24.037607 11664 hybrid_clock.cc:648] HybridClock initialized: now 1790376264037559 us; error 67 us; skew 500 ppm
Sep 25 22:44:24 dist-test-slave-5vdr krb5kdc[11557](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376264, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.198.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:44:24.041917 11664 init.cc:377] Logged in from keytab as kudu/127.10.198.1@KRBTEST.COM (short username kudu)
I20260925 22:44:24.043036 11664 webserver.cc:460] Webserver started at http://127.10.198.1:42269/ using document root <none> and password file <none>
I20260925 22:44:24.043687 11664 fs_manager.cc:359] Metadata directory not provided
I20260925 22:44:24.043777 11664 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:44:24.044126 11664 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:44:24.046540 11664 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790376259650642-11032-0/minicluster-data/ts-0/data/instance:
uuid: "1f265f2b8c5a422ba0c3e0f4d677918c"
format_stamp: "Formatted at 2026-09-25 22:44:24 on dist-test-slave-5vdr"
server_key: "c6bf47d6a6724635d7be6e416661b161"
server_key_iv: "059c3761c70d87f8d88f4d06f32a811b"
server_key_version: "encryptionkey@0"
I20260925 22:44:24.047304 11664 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790376259650642-11032-0/minicluster-data/ts-0/wal/instance:
uuid: "1f265f2b8c5a422ba0c3e0f4d677918c"
format_stamp: "Formatted at 2026-09-25 22:44:24 on dist-test-slave-5vdr"
server_key: "c6bf47d6a6724635d7be6e416661b161"
server_key_iv: "059c3761c70d87f8d88f4d06f32a811b"
server_key_version: "encryptionkey@0"
I20260925 22:44:24.051901 11664 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260925 22:44:24.055014 11679 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:44:24.056588 11664 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 22:44:24.056762 11664 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790376259650642-11032-0/minicluster-data/ts-0/data,/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790376259650642-11032-0/minicluster-data/ts-0/wal
uuid: "1f265f2b8c5a422ba0c3e0f4d677918c"
format_stamp: "Formatted at 2026-09-25 22:44:24 on dist-test-slave-5vdr"
server_key: "c6bf47d6a6724635d7be6e416661b161"
server_key_iv: "059c3761c70d87f8d88f4d06f32a811b"
server_key_version: "encryptionkey@0"
I20260925 22:44:24.056906 11664 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790376259650642-11032-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790376259650642-11032-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790376259650642-11032-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:44:24.071942 11664 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:44:24.075316 11664 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:44:24.075546 11664 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:44:24.076248 11664 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 22:44:24.077229 11664 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:44:24.077284 11664 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:44:24.077327 11664 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:44:24.077373 11664 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 22:44:24.088274 11664 rpc_server.cc:307] RPC server started. Bound to: 127.10.198.1:38669
I20260925 22:44:24.088313 11792 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.198.1:38669 every 8 connection(s)
I20260925 22:44:24.089406 11664 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790376259650642-11032-0/minicluster-data/ts-0/data/info.pb
I20260925 22:44:24.091969 11032 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrWXm8X/build/debug/bin/kudu as pid 11664
I20260925 22:44:24.092092 11032 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790376259650642-11032-0/minicluster-data/ts-0/wal/instance
I20260925 22:44:24.092387 11032 external_mini_cluster.cc:1428] Setting key ec956dfc8c586c1ffd94446b4c4b9b4b
Sep 25 22:44:24 dist-test-slave-5vdr krb5kdc[11557](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376264, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.198.1@KRBTEST.COM for kudu/127.10.198.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.10.198.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.198.2@KRBTEST.COM" created.
I20260925 22:44:24.108843 11641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:44:24.091596 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.198.1:56773 (local address 127.10.198.62:41253)
0925 22:44:24.091773 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0925 22:44:24.091777 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:44:24.093243 (+ 1466us) server_negotiation.cc:407] Connection header received
0925 22:44:24.094730 (+ 1487us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:44:24.094734 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:44:24.094799 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:44:24.094908 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:44:24.096796 (+ 1888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:44:24.097668 (+ 872us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:44:24.099987 (+ 2319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:44:24.100493 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:44:24.100661 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:44:24.104253 (+ 3592us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:44:24.104257 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:44:24.104259 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:44:24.104292 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:44:24.105982 (+ 1690us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:44:24.106824 (+ 842us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:44:24.106828 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:44:24.106829 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:44:24.106926 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:44:24.107433 (+ 507us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:44:24.107438 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:44:24.107443 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:44:24.107722 (+ 279us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:44:24.107850 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0925 22:44:24.108474 (+ 624us) server_negotiation.cc:299] Negotiation successful
0925 22:44:24.108654 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260925 22:44:24.110095 11795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:44:24.092130 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.198.62:41253 (local address 127.10.198.1:56773)
0925 22:44:24.092806 (+ 676us) negotiation.cc:107] Waiting for socket to connect
0925 22:44:24.092878 (+ 72us) client_negotiation.cc:174] Beginning negotiation
0925 22:44:24.094454 (+ 1576us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:44:24.095186 (+ 732us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:44:24.095197 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:44:24.095637 (+ 440us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:44:24.096556 (+ 919us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:44:24.096584 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:44:24.097891 (+ 1307us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:44:24.097897 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:44:24.099360 (+ 1463us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:44:24.099375 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:44:24.100719 (+ 1344us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:44:24.100722 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:44:24.100971 (+ 249us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:44:24.101717 (+ 746us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:44:24.101739 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:44:24.103988 (+ 2249us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:44:24.106156 (+ 2168us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:44:24.106163 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:44:24.106180 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:44:24.106662 (+ 482us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:44:24.107032 (+ 370us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:44:24.107038 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:44:24.107043 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:44:24.107291 (+ 248us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:44:24.107864 (+ 573us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:44:24.107873 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:44:24.108184 (+ 311us) client_negotiation.cc:769] Sending connection context
0925 22:44:24.108404 (+ 220us) client_negotiation.cc:240] Negotiation successful
0925 22:44:24.108823 (+ 419us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":408,"thread_start_us":183,"threads_started":1}
I20260925 22:44:24.111646 11793 heartbeater.cc:347] Connected to a master server at 127.10.198.62:41253
I20260925 22:44:24.112008 11793 heartbeater.cc:464] Registering TS with master...
I20260925 22:44:24.112596 11793 heartbeater.cc:511] Master 127.10.198.62:41253 requested a full tablet report, sending...
I20260925 22:44:24.114367 11602 ts_manager.cc:194] Registered new tserver with Master: 1f265f2b8c5a422ba0c3e0f4d677918c (127.10.198.1:38669)
I20260925 22:44:24.115588 11602 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.198.1@KRBTEST.COM'} at 127.10.198.1:56773
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.198.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790376259650642-11032-0/krb5kdc/kudu_127.10.198.2.keytab.
Entry for principal kudu/127.10.198.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790376259650642-11032-0/krb5kdc/kudu_127.10.198.2.keytab.
I20260925 22:44:24.124019 11032 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrWXm8X/build/debug/bin/kudu
/tmp/dist-test-taskrWXm8X/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790376259650642-11032-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790376259650642-11032-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790376259650642-11032-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790376259650642-11032-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.198.2:0
--local_ip_for_outbound_sockets=127.10.198.2
--webserver_interface=127.10.198.2
--webserver_port=0
--tserver_master_addrs=127.10.198.62:41253
--keytab_file=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790376259650642-11032-0/krb5kdc/kudu_127.10.198.2.keytab
--principal=kudu/127.10.198.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.198.20:36283
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790376259650642-11032-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790376259650642-11032-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790376259650642-11032-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:44:24.241366 11798 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:44:24.241683 11798 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:44:24.241731 11798 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:44:24.246058 11798 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:44:24.246146 11798 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:44:24.246264 11798 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.198.2
I20260925 22:44:24.252242 11798 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.198.20:36283
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790376259650642-11032-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790376259650642-11032-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790376259650642-11032-0/krb5kdc/kudu_127.10.198.2.keytab
--principal=kudu/127.10.198.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.198.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790376259650642-11032-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.198.2
--webserver_port=0
--tserver_master_addrs=127.10.198.62:41253
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11798
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.198.2
--log_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790376259650642-11032-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:09:16 UTC on ccycloud.quasar-afmhyw.root.comops.site
build id 4673379
I20260925 22:44:24.253412 11798 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:44:24.254496 11798 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:44:24.265317 11803 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:44:24.265322 11804 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:44:24.265322 11806 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:44:24.265731 11798 server_base.cc:1034] running on GCE node
I20260925 22:44:24.266255 11798 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:44:24.267016 11798 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:44:24.268188 11798 hybrid_clock.cc:648] HybridClock initialized: now 1790376264268167 us; error 57 us; skew 500 ppm
Sep 25 22:44:24 dist-test-slave-5vdr krb5kdc[11557](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376264, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.198.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:44:24.271279 11798 init.cc:377] Logged in from keytab as kudu/127.10.198.2@KRBTEST.COM (short username kudu)
I20260925 22:44:24.272493 11798 webserver.cc:460] Webserver started at http://127.10.198.2:41637/ using document root <none> and password file <none>
I20260925 22:44:24.273108 11798 fs_manager.cc:359] Metadata directory not provided
I20260925 22:44:24.273183 11798 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:44:24.273367 11798 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:44:24.275660 11798 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790376259650642-11032-0/minicluster-data/ts-1/data/instance:
uuid: "276843c81a0e4c7c998527f76b02145d"
format_stamp: "Formatted at 2026-09-25 22:44:24 on dist-test-slave-5vdr"
server_key: "56c29039555e839a24b520cd106ea957"
server_key_iv: "b7040f6f03e731108a896f2fa4023d96"
server_key_version: "encryptionkey@0"
I20260925 22:44:24.276384 11798 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790376259650642-11032-0/minicluster-data/ts-1/wal/instance:
uuid: "276843c81a0e4c7c998527f76b02145d"
format_stamp: "Formatted at 2026-09-25 22:44:24 on dist-test-slave-5vdr"
server_key: "56c29039555e839a24b520cd106ea957"
server_key_iv: "b7040f6f03e731108a896f2fa4023d96"
server_key_version: "encryptionkey@0"
I20260925 22:44:24.280946 11798 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260925 22:44:24.283985 11813 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:44:24.285470 11798 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 22:44:24.285651 11798 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790376259650642-11032-0/minicluster-data/ts-1/data,/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790376259650642-11032-0/minicluster-data/ts-1/wal
uuid: "276843c81a0e4c7c998527f76b02145d"
format_stamp: "Formatted at 2026-09-25 22:44:24 on dist-test-slave-5vdr"
server_key: "56c29039555e839a24b520cd106ea957"
server_key_iv: "b7040f6f03e731108a896f2fa4023d96"
server_key_version: "encryptionkey@0"
I20260925 22:44:24.285799 11798 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790376259650642-11032-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790376259650642-11032-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790376259650642-11032-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:44:24.302791 11798 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:44:24.306288 11798 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:44:24.306521 11798 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:44:24.307156 11798 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 22:44:24.308133 11798 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:44:24.308205 11798 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:44:24.308280 11798 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:44:24.308331 11798 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 22:44:24.318825 11798 rpc_server.cc:307] RPC server started. Bound to: 127.10.198.2:38273
I20260925 22:44:24.318845 11926 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.198.2:38273 every 8 connection(s)
I20260925 22:44:24.319975 11798 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790376259650642-11032-0/minicluster-data/ts-1/data/info.pb
I20260925 22:44:24.321231 11032 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrWXm8X/build/debug/bin/kudu as pid 11798
I20260925 22:44:24.321341 11032 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790376259650642-11032-0/minicluster-data/ts-1/wal/instance
I20260925 22:44:24.321600 11032 external_mini_cluster.cc:1428] Setting key 7ce8ba137f74a9b00e9f0ae73a44837d
WARNING: no policy specified for kudu/127.10.198.3@KRBTEST.COM; defaulting to no policy
Sep 25 22:44:24 dist-test-slave-5vdr krb5kdc[11557](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376264, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.198.2@KRBTEST.COM for kudu/127.10.198.62@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.198.3@KRBTEST.COM" created.
I20260925 22:44:24.337608 11641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:44:24.322656 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.198.2:53109 (local address 127.10.198.62:41253)
0925 22:44:24.322808 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0925 22:44:24.322813 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:44:24.323730 (+ 917us) server_negotiation.cc:407] Connection header received
0925 22:44:24.325271 (+ 1541us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:44:24.325277 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:44:24.325356 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:44:24.325475 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:44:24.326924 (+ 1449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:44:24.327875 (+ 951us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:44:24.329195 (+ 1320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:44:24.329601 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:44:24.329738 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:44:24.332752 (+ 3014us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:44:24.332756 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:44:24.332758 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:44:24.332788 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:44:24.334688 (+ 1900us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:44:24.335580 (+ 892us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:44:24.335583 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:44:24.335585 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:44:24.335682 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:44:24.336183 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:44:24.336186 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:44:24.336187 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:44:24.336399 (+ 212us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:44:24.336485 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0925 22:44:24.337204 (+ 719us) server_negotiation.cc:299] Negotiation successful
0925 22:44:24.337387 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260925 22:44:24.338399 11929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:44:24.322864 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.198.62:41253 (local address 127.10.198.2:53109)
0925 22:44:24.323519 (+ 655us) negotiation.cc:107] Waiting for socket to connect
0925 22:44:24.323584 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0925 22:44:24.324948 (+ 1364us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:44:24.325704 (+ 756us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:44:24.325712 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:44:24.326128 (+ 416us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:44:24.326689 (+ 561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:44:24.326706 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:44:24.328019 (+ 1313us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:44:24.328022 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:44:24.329031 (+ 1009us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:44:24.329043 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:44:24.329781 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:44:24.329784 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:44:24.329989 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:44:24.330685 (+ 696us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:44:24.330704 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:44:24.332511 (+ 1807us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:44:24.334901 (+ 2390us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:44:24.334911 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:44:24.334935 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:44:24.335389 (+ 454us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:44:24.335795 (+ 406us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:44:24.335802 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:44:24.335807 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:44:24.336074 (+ 267us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:44:24.336514 (+ 440us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:44:24.336525 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:44:24.336903 (+ 378us) client_negotiation.cc:769] Sending connection context
0925 22:44:24.337086 (+ 183us) client_negotiation.cc:240] Negotiation successful
0925 22:44:24.337384 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":399,"thread_start_us":194,"threads_started":1}
I20260925 22:44:24.339825 11927 heartbeater.cc:347] Connected to a master server at 127.10.198.62:41253
I20260925 22:44:24.340170 11927 heartbeater.cc:464] Registering TS with master...
I20260925 22:44:24.340937 11927 heartbeater.cc:511] Master 127.10.198.62:41253 requested a full tablet report, sending...
I20260925 22:44:24.342314 11602 ts_manager.cc:194] Registered new tserver with Master: 276843c81a0e4c7c998527f76b02145d (127.10.198.2:38273)
I20260925 22:44:24.342941 11602 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.198.2@KRBTEST.COM'} at 127.10.198.2:53109
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.198.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790376259650642-11032-0/krb5kdc/kudu_127.10.198.3.keytab.
Entry for principal kudu/127.10.198.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790376259650642-11032-0/krb5kdc/kudu_127.10.198.3.keytab.
I20260925 22:44:24.353288 11032 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrWXm8X/build/debug/bin/kudu
/tmp/dist-test-taskrWXm8X/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790376259650642-11032-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790376259650642-11032-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790376259650642-11032-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790376259650642-11032-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.198.3:0
--local_ip_for_outbound_sockets=127.10.198.3
--webserver_interface=127.10.198.3
--webserver_port=0
--tserver_master_addrs=127.10.198.62:41253
--keytab_file=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790376259650642-11032-0/krb5kdc/kudu_127.10.198.3.keytab
--principal=kudu/127.10.198.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.198.20:36283
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790376259650642-11032-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790376259650642-11032-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790376259650642-11032-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:44:24.473605 11932 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:44:24.473984 11932 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:44:24.474071 11932 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:44:24.478417 11932 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:44:24.478559 11932 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:44:24.478665 11932 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.198.3
I20260925 22:44:24.484608 11932 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.198.20:36283
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790376259650642-11032-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790376259650642-11032-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790376259650642-11032-0/krb5kdc/kudu_127.10.198.3.keytab
--principal=kudu/127.10.198.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.198.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790376259650642-11032-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.198.3
--webserver_port=0
--tserver_master_addrs=127.10.198.62:41253
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11932
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.198.3
--log_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790376259650642-11032-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:09:16 UTC on ccycloud.quasar-afmhyw.root.comops.site
build id 4673379
I20260925 22:44:24.485850 11932 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:44:24.487035 11932 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:44:24.498025 11937 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:44:24.498000 11940 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:44:24.498131 11938 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:44:24.498586 11932 server_base.cc:1034] running on GCE node
I20260925 22:44:24.499156 11932 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:44:24.499984 11932 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:44:24.501289 11932 hybrid_clock.cc:648] HybridClock initialized: now 1790376264501173 us; error 135 us; skew 500 ppm
Sep 25 22:44:24 dist-test-slave-5vdr krb5kdc[11557](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376264, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.198.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:44:24.504891 11932 init.cc:377] Logged in from keytab as kudu/127.10.198.3@KRBTEST.COM (short username kudu)
I20260925 22:44:24.506378 11932 webserver.cc:460] Webserver started at http://127.10.198.3:36487/ using document root <none> and password file <none>
I20260925 22:44:24.507105 11932 fs_manager.cc:359] Metadata directory not provided
I20260925 22:44:24.507221 11932 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:44:24.507491 11932 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:44:24.510191 11932 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790376259650642-11032-0/minicluster-data/ts-2/data/instance:
uuid: "4424247be5514d9b98cbdbb48b861d46"
format_stamp: "Formatted at 2026-09-25 22:44:24 on dist-test-slave-5vdr"
server_key: "8cf6ce471eee3b77348849de8bd2a672"
server_key_iv: "f24ca54f79be1b493ecc72848f29526d"
server_key_version: "encryptionkey@0"
I20260925 22:44:24.510917 11932 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790376259650642-11032-0/minicluster-data/ts-2/wal/instance:
uuid: "4424247be5514d9b98cbdbb48b861d46"
format_stamp: "Formatted at 2026-09-25 22:44:24 on dist-test-slave-5vdr"
server_key: "8cf6ce471eee3b77348849de8bd2a672"
server_key_iv: "f24ca54f79be1b493ecc72848f29526d"
server_key_version: "encryptionkey@0"
I20260925 22:44:24.515785 11932 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260925 22:44:24.519627 11947 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:44:24.521358 11932 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 22:44:24.521566 11932 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790376259650642-11032-0/minicluster-data/ts-2/data,/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790376259650642-11032-0/minicluster-data/ts-2/wal
uuid: "4424247be5514d9b98cbdbb48b861d46"
format_stamp: "Formatted at 2026-09-25 22:44:24 on dist-test-slave-5vdr"
server_key: "8cf6ce471eee3b77348849de8bd2a672"
server_key_iv: "f24ca54f79be1b493ecc72848f29526d"
server_key_version: "encryptionkey@0"
I20260925 22:44:24.521723 11932 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790376259650642-11032-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790376259650642-11032-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790376259650642-11032-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:44:24.542284 11932 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:44:24.546687 11932 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:44:24.546952 11932 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:44:24.547686 11932 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 22:44:24.548840 11932 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:44:24.548929 11932 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:44:24.549003 11932 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:44:24.549057 11932 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 22:44:24.560132 11932 rpc_server.cc:307] RPC server started. Bound to: 127.10.198.3:32859
I20260925 22:44:24.560189 12060 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.198.3:32859 every 8 connection(s)
I20260925 22:44:24.561551 11932 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790376259650642-11032-0/minicluster-data/ts-2/data/info.pb
I20260925 22:44:24.571435 11032 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrWXm8X/build/debug/bin/kudu as pid 11932
I20260925 22:44:24.571687 11032 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790376259650642-11032-0/minicluster-data/ts-2/wal/instance
I20260925 22:44:24.572073 11032 external_mini_cluster.cc:1428] Setting key a6dce46d34c4115d1ea263f4a1f88c58
Sep 25 22:44:24 dist-test-slave-5vdr krb5kdc[11557](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376264, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.198.3@KRBTEST.COM for kudu/127.10.198.62@KRBTEST.COM
I20260925 22:44:24.578044 11641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:44:24.563909 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.198.3:52935 (local address 127.10.198.62:41253)
0925 22:44:24.564061 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0925 22:44:24.564066 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:44:24.565287 (+ 1221us) server_negotiation.cc:407] Connection header received
0925 22:44:24.566340 (+ 1053us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:44:24.566344 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:44:24.566403 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:44:24.566493 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:44:24.568066 (+ 1573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:44:24.568623 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:44:24.570083 (+ 1460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:44:24.570346 (+ 263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:44:24.570458 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:44:24.573850 (+ 3392us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:44:24.573854 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:44:24.573857 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:44:24.573897 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:44:24.575478 (+ 1581us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:44:24.576250 (+ 772us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:44:24.576254 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:44:24.576255 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:44:24.576350 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:44:24.576770 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:44:24.576778 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:44:24.576779 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:44:24.576991 (+ 212us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:44:24.577132 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0925 22:44:24.577725 (+ 593us) server_negotiation.cc:299] Negotiation successful
0925 22:44:24.577871 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260925 22:44:24.578979 12063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:44:24.564452 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.198.62:41253 (local address 127.10.198.3:52935)
0925 22:44:24.565066 (+ 614us) negotiation.cc:107] Waiting for socket to connect
0925 22:44:24.565123 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0925 22:44:24.566052 (+ 929us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:44:24.566746 (+ 694us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:44:24.566755 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:44:24.567185 (+ 430us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:44:24.567781 (+ 596us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:44:24.567797 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:44:24.568823 (+ 1026us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:44:24.568828 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:44:24.569889 (+ 1061us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:44:24.569904 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:44:24.570565 (+ 661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:44:24.570568 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:44:24.570770 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:44:24.571618 (+ 848us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:44:24.571652 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:44:24.573661 (+ 2009us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:44:24.575640 (+ 1979us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:44:24.575648 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:44:24.575677 (+ 29us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:44:24.576105 (+ 428us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:44:24.576465 (+ 360us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:44:24.576470 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:44:24.576472 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:44:24.576649 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:44:24.577131 (+ 482us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:44:24.577140 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:44:24.577447 (+ 307us) client_negotiation.cc:769] Sending connection context
0925 22:44:24.577657 (+ 210us) client_negotiation.cc:240] Negotiation successful
0925 22:44:24.578054 (+ 397us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":351,"thread_start_us":151,"threads_started":1}
I20260925 22:44:24.581123 12061 heartbeater.cc:347] Connected to a master server at 127.10.198.62:41253
I20260925 22:44:24.581450 12061 heartbeater.cc:464] Registering TS with master...
I20260925 22:44:24.582084 12061 heartbeater.cc:511] Master 127.10.198.62:41253 requested a full tablet report, sending...
I20260925 22:44:24.583395 11602 ts_manager.cc:194] Registered new tserver with Master: 4424247be5514d9b98cbdbb48b861d46 (127.10.198.3:32859)
I20260925 22:44:24.584067 11602 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.198.3@KRBTEST.COM'} at 127.10.198.3:52935
I20260925 22:44:24.587162 11032 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 22:44:24.601094 11641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:44:24.591508 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39032 (local address 127.10.198.62:41253)
0925 22:44:24.591748 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0925 22:44:24.591753 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:44:24.592007 (+ 254us) server_negotiation.cc:407] Connection header received
0925 22:44:24.592193 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:44:24.592196 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:44:24.592243 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:44:24.592347 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:44:24.593116 (+ 769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:44:24.593649 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:44:24.594529 (+ 880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:44:24.594765 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:44:24.594889 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:44:24.596825 (+ 1936us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:44:24.596828 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:44:24.596830 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:44:24.596864 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:44:24.598829 (+ 1965us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:44:24.599466 (+ 637us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:44:24.599470 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:44:24.599472 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:44:24.599577 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:44:24.599975 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:44:24.599979 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:44:24.599980 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:44:24.600350 (+ 370us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:44:24.600514 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0925 22:44:24.600787 (+ 273us) server_negotiation.cc:299] Negotiation successful
0925 22:44:24.600917 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260925 22:44:24.606751 11602 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39032:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 22:44:24.610073 11602 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 22:44:24.627045 12077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:44:24.620807 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57798 (local address 127.10.198.3:32859)
0925 22:44:24.621236 (+ 429us) server_negotiation.cc:206] Beginning negotiation
0925 22:44:24.621250 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0925 22:44:24.621846 (+ 596us) server_negotiation.cc:407] Connection header received
0925 22:44:24.621926 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:44:24.621935 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:44:24.622115 (+ 180us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:44:24.622258 (+ 143us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:44:24.622655 (+ 397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:44:24.623823 (+ 1168us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:44:24.625728 (+ 1905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:44:24.626383 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:44:24.626492 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:44:24.626637 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0925 22:44:24.626760 (+ 123us) server_negotiation.cc:299] Negotiation successful
0925 22:44:24.626866 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":330,"thread_start_us":129,"threads_started":1}
I20260925 22:44:24.627097 12074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:44:24.619990 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42292 (local address 127.10.198.2:38273)
0925 22:44:24.620445 (+ 455us) server_negotiation.cc:206] Beginning negotiation
0925 22:44:24.620459 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0925 22:44:24.621290 (+ 831us) server_negotiation.cc:407] Connection header received
0925 22:44:24.621639 (+ 349us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:44:24.621647 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:44:24.621874 (+ 227us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:44:24.621981 (+ 107us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:44:24.622656 (+ 675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:44:24.623428 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:44:24.625057 (+ 1629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:44:24.626002 (+ 945us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:44:24.626120 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:44:24.626292 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0925 22:44:24.626801 (+ 509us) server_negotiation.cc:299] Negotiation successful
0925 22:44:24.626918 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":343,"thread_start_us":108,"threads_started":1}
I20260925 22:44:24.627230 12075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:44:24.619984 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.198.1:38669 (local address 127.0.0.1:40450)
0925 22:44:24.621323 (+ 1339us) negotiation.cc:107] Waiting for socket to connect
0925 22:44:24.621337 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 22:44:24.621444 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:44:24.622174 (+ 730us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:44:24.622180 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:44:24.622202 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:44:24.622482 (+ 280us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:44:24.622489 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:44:24.624012 (+ 1523us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:44:24.624015 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:44:24.625391 (+ 1376us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:44:24.625401 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:44:24.626772 (+ 1371us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:44:24.626776 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:44:24.626875 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:44:24.626896 (+ 21us) client_negotiation.cc:769] Sending connection context
0925 22:44:24.626964 (+ 68us) client_negotiation.cc:240] Negotiation successful
0925 22:44:24.627045 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1276,"mutex_wait_us":151,"thread_start_us":123,"threads_started":1}
I20260925 22:44:24.627480 12076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:44:24.620687 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40450 (local address 127.10.198.1:38669)
0925 22:44:24.621050 (+ 363us) server_negotiation.cc:206] Beginning negotiation
0925 22:44:24.621056 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 22:44:24.621447 (+ 391us) server_negotiation.cc:407] Connection header received
0925 22:44:24.621643 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:44:24.621649 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:44:24.621883 (+ 234us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:44:24.621999 (+ 116us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:44:24.622608 (+ 609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:44:24.623824 (+ 1216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:44:24.625508 (+ 1684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:44:24.626356 (+ 848us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:44:24.626553 (+ 197us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:44:24.626706 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0925 22:44:24.627127 (+ 421us) server_negotiation.cc:299] Negotiation successful
0925 22:44:24.627328 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":276,"thread_start_us":83,"threads_started":1}
I20260925 22:44:24.627040 12072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:44:24.620259 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.198.3:32859 (local address 127.0.0.1:57798)
0925 22:44:24.621654 (+ 1395us) negotiation.cc:107] Waiting for socket to connect
0925 22:44:24.621667 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 22:44:24.621741 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:44:24.622257 (+ 516us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:44:24.622260 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:44:24.622273 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:44:24.622514 (+ 241us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:44:24.622520 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:44:24.624012 (+ 1492us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:44:24.624015 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:44:24.625603 (+ 1588us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:44:24.625610 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:44:24.626513 (+ 903us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:44:24.626517 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:44:24.626608 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:44:24.626631 (+ 23us) client_negotiation.cc:769] Sending connection context
0925 22:44:24.626729 (+ 98us) client_negotiation.cc:240] Negotiation successful
0925 22:44:24.626878 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1338,"mutex_wait_us":501,"thread_start_us":81,"threads_started":1}
I20260925 22:44:24.628050 12073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:44:24.619854 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.198.2:38273 (local address 127.0.0.1:42292)
0925 22:44:24.621124 (+ 1270us) negotiation.cc:107] Waiting for socket to connect
0925 22:44:24.621194 (+ 70us) client_negotiation.cc:174] Beginning negotiation
0925 22:44:24.621439 (+ 245us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:44:24.622160 (+ 721us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:44:24.622165 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:44:24.622183 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:44:24.622512 (+ 329us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:44:24.622520 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:44:24.623576 (+ 1056us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:44:24.623581 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:44:24.624928 (+ 1347us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:44:24.624938 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:44:24.626387 (+ 1449us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:44:24.626393 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:44:24.626555 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:44:24.626606 (+ 51us) client_negotiation.cc:769] Sending connection context
0925 22:44:24.627799 (+ 1193us) client_negotiation.cc:240] Negotiation successful
0925 22:44:24.627918 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1067,"mutex_wait_us":66,"thread_start_us":117,"threads_started":1}
I20260925 22:44:24.630689 11995 tablet_service.cc:1467] Processing CreateTablet for tablet e602722e9856465fbc3769d7752deaca (DEFAULT_TABLE table=test-table [id=aa06f211563f49748f29afd0bb5e4fde]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 22:44:24.630823 11861 tablet_service.cc:1467] Processing CreateTablet for tablet e602722e9856465fbc3769d7752deaca (DEFAULT_TABLE table=test-table [id=aa06f211563f49748f29afd0bb5e4fde]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 22:44:24.630796 11727 tablet_service.cc:1467] Processing CreateTablet for tablet e602722e9856465fbc3769d7752deaca (DEFAULT_TABLE table=test-table [id=aa06f211563f49748f29afd0bb5e4fde]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 22:44:24.632118 11995 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e602722e9856465fbc3769d7752deaca. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:44:24.632118 11861 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e602722e9856465fbc3769d7752deaca. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:44:24.632118 11727 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e602722e9856465fbc3769d7752deaca. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:44:24.639494 12078 tablet_bootstrap.cc:492] T e602722e9856465fbc3769d7752deaca P 4424247be5514d9b98cbdbb48b861d46: Bootstrap starting.
I20260925 22:44:24.641412 12079 tablet_bootstrap.cc:492] T e602722e9856465fbc3769d7752deaca P 1f265f2b8c5a422ba0c3e0f4d677918c: Bootstrap starting.
I20260925 22:44:24.642139 12078 tablet_bootstrap.cc:654] T e602722e9856465fbc3769d7752deaca P 4424247be5514d9b98cbdbb48b861d46: Neither blocks nor log segments found. Creating new log.
I20260925 22:44:24.643352 12078 log.cc:824] T e602722e9856465fbc3769d7752deaca P 4424247be5514d9b98cbdbb48b861d46: Log is configured to *not* fsync() on all Append() calls
I20260925 22:44:24.643683 12079 tablet_bootstrap.cc:654] T e602722e9856465fbc3769d7752deaca P 1f265f2b8c5a422ba0c3e0f4d677918c: Neither blocks nor log segments found. Creating new log.
I20260925 22:44:24.644212 12080 tablet_bootstrap.cc:492] T e602722e9856465fbc3769d7752deaca P 276843c81a0e4c7c998527f76b02145d: Bootstrap starting.
I20260925 22:44:24.645565 12078 tablet_bootstrap.cc:492] T e602722e9856465fbc3769d7752deaca P 4424247be5514d9b98cbdbb48b861d46: No bootstrap required, opened a new log
I20260925 22:44:24.645823 12078 ts_tablet_manager.cc:1397] T e602722e9856465fbc3769d7752deaca P 4424247be5514d9b98cbdbb48b861d46: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260925 22:44:24.646592 12080 tablet_bootstrap.cc:654] T e602722e9856465fbc3769d7752deaca P 276843c81a0e4c7c998527f76b02145d: Neither blocks nor log segments found. Creating new log.
I20260925 22:44:24.646649 12079 log.cc:824] T e602722e9856465fbc3769d7752deaca P 1f265f2b8c5a422ba0c3e0f4d677918c: Log is configured to *not* fsync() on all Append() calls
I20260925 22:44:24.647449 12080 log.cc:824] T e602722e9856465fbc3769d7752deaca P 276843c81a0e4c7c998527f76b02145d: Log is configured to *not* fsync() on all Append() calls
I20260925 22:44:24.649101 12079 tablet_bootstrap.cc:492] T e602722e9856465fbc3769d7752deaca P 1f265f2b8c5a422ba0c3e0f4d677918c: No bootstrap required, opened a new log
I20260925 22:44:24.649343 12079 ts_tablet_manager.cc:1397] T e602722e9856465fbc3769d7752deaca P 1f265f2b8c5a422ba0c3e0f4d677918c: Time spent bootstrapping tablet: real 0.008s user 0.004s sys 0.000s
I20260925 22:44:24.649674 12080 tablet_bootstrap.cc:492] T e602722e9856465fbc3769d7752deaca P 276843c81a0e4c7c998527f76b02145d: No bootstrap required, opened a new log
I20260925 22:44:24.649878 12080 ts_tablet_manager.cc:1397] T e602722e9856465fbc3769d7752deaca P 276843c81a0e4c7c998527f76b02145d: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260925 22:44:24.649883 12078 raft_consensus.cc:348] T e602722e9856465fbc3769d7752deaca P 4424247be5514d9b98cbdbb48b861d46 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "276843c81a0e4c7c998527f76b02145d" member_type: VOTER last_known_addr { host: "127.10.198.2" port: 38273 } } peers { permanent_uuid: "1f265f2b8c5a422ba0c3e0f4d677918c" member_type: VOTER last_known_addr { host: "127.10.198.1" port: 38669 } } peers { permanent_uuid: "4424247be5514d9b98cbdbb48b861d46" member_type: VOTER last_known_addr { host: "127.10.198.3" port: 32859 } }
I20260925 22:44:24.650223 12078 raft_consensus.cc:374] T e602722e9856465fbc3769d7752deaca P 4424247be5514d9b98cbdbb48b861d46 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:44:24.650283 12078 raft_consensus.cc:729] T e602722e9856465fbc3769d7752deaca P 4424247be5514d9b98cbdbb48b861d46 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4424247be5514d9b98cbdbb48b861d46, State: Initialized, Role: FOLLOWER
I20260925 22:44:24.650830 12078 consensus_queue.cc:260] T e602722e9856465fbc3769d7752deaca P 4424247be5514d9b98cbdbb48b861d46 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "276843c81a0e4c7c998527f76b02145d" member_type: VOTER last_known_addr { host: "127.10.198.2" port: 38273 } } peers { permanent_uuid: "1f265f2b8c5a422ba0c3e0f4d677918c" member_type: VOTER last_known_addr { host: "127.10.198.1" port: 38669 } } peers { permanent_uuid: "4424247be5514d9b98cbdbb48b861d46" member_type: VOTER last_known_addr { host: "127.10.198.3" port: 32859 } }
I20260925 22:44:24.652140 12078 ts_tablet_manager.cc:1428] T e602722e9856465fbc3769d7752deaca P 4424247be5514d9b98cbdbb48b861d46: Time spent starting tablet: real 0.006s user 0.008s sys 0.000s
I20260925 22:44:24.652407 12061 heartbeater.cc:503] Master 127.10.198.62:41253 was elected leader, sending a full tablet report...
I20260925 22:44:24.654323 12079 raft_consensus.cc:348] T e602722e9856465fbc3769d7752deaca P 1f265f2b8c5a422ba0c3e0f4d677918c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "276843c81a0e4c7c998527f76b02145d" member_type: VOTER last_known_addr { host: "127.10.198.2" port: 38273 } } peers { permanent_uuid: "1f265f2b8c5a422ba0c3e0f4d677918c" member_type: VOTER last_known_addr { host: "127.10.198.1" port: 38669 } } peers { permanent_uuid: "4424247be5514d9b98cbdbb48b861d46" member_type: VOTER last_known_addr { host: "127.10.198.3" port: 32859 } }
I20260925 22:44:24.654323 12080 raft_consensus.cc:348] T e602722e9856465fbc3769d7752deaca P 276843c81a0e4c7c998527f76b02145d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "276843c81a0e4c7c998527f76b02145d" member_type: VOTER last_known_addr { host: "127.10.198.2" port: 38273 } } peers { permanent_uuid: "1f265f2b8c5a422ba0c3e0f4d677918c" member_type: VOTER last_known_addr { host: "127.10.198.1" port: 38669 } } peers { permanent_uuid: "4424247be5514d9b98cbdbb48b861d46" member_type: VOTER last_known_addr { host: "127.10.198.3" port: 32859 } }
I20260925 22:44:24.654660 12079 raft_consensus.cc:374] T e602722e9856465fbc3769d7752deaca P 1f265f2b8c5a422ba0c3e0f4d677918c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:44:24.654660 12080 raft_consensus.cc:374] T e602722e9856465fbc3769d7752deaca P 276843c81a0e4c7c998527f76b02145d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:44:24.654714 12080 raft_consensus.cc:729] T e602722e9856465fbc3769d7752deaca P 276843c81a0e4c7c998527f76b02145d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 276843c81a0e4c7c998527f76b02145d, State: Initialized, Role: FOLLOWER
I20260925 22:44:24.654714 12079 raft_consensus.cc:729] T e602722e9856465fbc3769d7752deaca P 1f265f2b8c5a422ba0c3e0f4d677918c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f265f2b8c5a422ba0c3e0f4d677918c, State: Initialized, Role: FOLLOWER
I20260925 22:44:24.655246 12080 consensus_queue.cc:260] T e602722e9856465fbc3769d7752deaca P 276843c81a0e4c7c998527f76b02145d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "276843c81a0e4c7c998527f76b02145d" member_type: VOTER last_known_addr { host: "127.10.198.2" port: 38273 } } peers { permanent_uuid: "1f265f2b8c5a422ba0c3e0f4d677918c" member_type: VOTER last_known_addr { host: "127.10.198.1" port: 38669 } } peers { permanent_uuid: "4424247be5514d9b98cbdbb48b861d46" member_type: VOTER last_known_addr { host: "127.10.198.3" port: 32859 } }
I20260925 22:44:24.655246 12079 consensus_queue.cc:260] T e602722e9856465fbc3769d7752deaca P 1f265f2b8c5a422ba0c3e0f4d677918c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "276843c81a0e4c7c998527f76b02145d" member_type: VOTER last_known_addr { host: "127.10.198.2" port: 38273 } } peers { permanent_uuid: "1f265f2b8c5a422ba0c3e0f4d677918c" member_type: VOTER last_known_addr { host: "127.10.198.1" port: 38669 } } peers { permanent_uuid: "4424247be5514d9b98cbdbb48b861d46" member_type: VOTER last_known_addr { host: "127.10.198.3" port: 32859 } }
I20260925 22:44:24.656500 12079 ts_tablet_manager.cc:1428] T e602722e9856465fbc3769d7752deaca P 1f265f2b8c5a422ba0c3e0f4d677918c: Time spent starting tablet: real 0.007s user 0.008s sys 0.000s
I20260925 22:44:24.656545 11927 heartbeater.cc:503] Master 127.10.198.62:41253 was elected leader, sending a full tablet report...
I20260925 22:44:24.656625 12080 ts_tablet_manager.cc:1428] T e602722e9856465fbc3769d7752deaca P 276843c81a0e4c7c998527f76b02145d: Time spent starting tablet: real 0.007s user 0.008s sys 0.000s
I20260925 22:44:24.657804 11793 heartbeater.cc:503] Master 127.10.198.62:41253 was elected leader, sending a full tablet report...
I20260925 22:44:24.694718 12086 raft_consensus.cc:482] T e602722e9856465fbc3769d7752deaca P 1f265f2b8c5a422ba0c3e0f4d677918c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 22:44:24.694959 12086 raft_consensus.cc:504] T e602722e9856465fbc3769d7752deaca P 1f265f2b8c5a422ba0c3e0f4d677918c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "276843c81a0e4c7c998527f76b02145d" member_type: VOTER last_known_addr { host: "127.10.198.2" port: 38273 } } peers { permanent_uuid: "1f265f2b8c5a422ba0c3e0f4d677918c" member_type: VOTER last_known_addr { host: "127.10.198.1" port: 38669 } } peers { permanent_uuid: "4424247be5514d9b98cbdbb48b861d46" member_type: VOTER last_known_addr { host: "127.10.198.3" port: 32859 } }
I20260925 22:44:24.696277 12086 leader_election.cc:288] T e602722e9856465fbc3769d7752deaca P 1f265f2b8c5a422ba0c3e0f4d677918c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 276843c81a0e4c7c998527f76b02145d (127.10.198.2:38273), 4424247be5514d9b98cbdbb48b861d46 (127.10.198.3:32859)
I20260925 22:44:24.701015 12074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:44:24.696640 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.198.1:48439 (local address 127.10.198.2:38273)
0925 22:44:24.696880 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0925 22:44:24.696885 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:44:24.696978 (+ 93us) server_negotiation.cc:407] Connection header received
0925 22:44:24.697126 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:44:24.697132 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:44:24.697193 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:44:24.697269 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:44:24.697700 (+ 431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:44:24.698283 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:44:24.699566 (+ 1283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:44:24.700225 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:44:24.700366 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:44:24.700424 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0925 22:44:24.700761 (+ 337us) server_negotiation.cc:299] Negotiation successful
0925 22:44:24.700869 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":144}
I20260925 22:44:24.701076 12088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:44:24.696591 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.198.3:32859 (local address 127.10.198.1:48939)
0925 22:44:24.697069 (+ 478us) negotiation.cc:107] Waiting for socket to connect
0925 22:44:24.697085 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 22:44:24.697190 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:44:24.697481 (+ 291us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:44:24.697483 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:44:24.697498 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:44:24.697731 (+ 233us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:44:24.697739 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:44:24.698510 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:44:24.698513 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:44:24.699364 (+ 851us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:44:24.699375 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:44:24.700537 (+ 1162us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:44:24.700540 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:44:24.700678 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:44:24.700707 (+ 29us) client_negotiation.cc:769] Sending connection context
0925 22:44:24.700832 (+ 125us) client_negotiation.cc:240] Negotiation successful
0925 22:44:24.700905 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":386,"mutex_wait_us":47,"thread_start_us":129,"threads_started":1}
I20260925 22:44:24.701107 12077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:44:24.696708 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.198.1:48939 (local address 127.10.198.3:32859)
0925 22:44:24.696890 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0925 22:44:24.696898 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 22:44:24.697178 (+ 280us) server_negotiation.cc:407] Connection header received
0925 22:44:24.697308 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:44:24.697311 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:44:24.697362 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:44:24.697446 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:44:24.697854 (+ 408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:44:24.698390 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:44:24.699520 (+ 1130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:44:24.700225 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:44:24.700366 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:44:24.700424 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0925 22:44:24.700877 (+ 453us) server_negotiation.cc:299] Negotiation successful
0925 22:44:24.700992 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":96}
I20260925 22:44:24.701081 12087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:44:24.696509 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.198.2:38273 (local address 127.10.198.1:48439)
0925 22:44:24.696851 (+ 342us) negotiation.cc:107] Waiting for socket to connect
0925 22:44:24.696868 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 22:44:24.697009 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:44:24.697299 (+ 290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:44:24.697302 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:44:24.697320 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:44:24.697467 (+ 147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:44:24.697474 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:44:24.698411 (+ 937us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:44:24.698417 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:44:24.699341 (+ 924us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:44:24.699381 (+ 40us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:44:24.700366 (+ 985us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:44:24.700373 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:44:24.700586 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:44:24.700615 (+ 29us) client_negotiation.cc:769] Sending connection context
0925 22:44:24.700720 (+ 105us) client_negotiation.cc:240] Negotiation successful
0925 22:44:24.700815 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":265,"mutex_wait_us":21,"thread_start_us":147,"threads_started":1}
I20260925 22:44:24.701864 12015 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e602722e9856465fbc3769d7752deaca" candidate_uuid: "1f265f2b8c5a422ba0c3e0f4d677918c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4424247be5514d9b98cbdbb48b861d46" is_pre_election: true
I20260925 22:44:24.702170 11881 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e602722e9856465fbc3769d7752deaca" candidate_uuid: "1f265f2b8c5a422ba0c3e0f4d677918c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "276843c81a0e4c7c998527f76b02145d" is_pre_election: true
I20260925 22:44:24.702243 12015 raft_consensus.cc:2454] T e602722e9856465fbc3769d7752deaca P 4424247be5514d9b98cbdbb48b861d46 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1f265f2b8c5a422ba0c3e0f4d677918c in term 0.
I20260925 22:44:24.702467 11881 raft_consensus.cc:2454] T e602722e9856465fbc3769d7752deaca P 276843c81a0e4c7c998527f76b02145d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1f265f2b8c5a422ba0c3e0f4d677918c in term 0.
I20260925 22:44:24.702833 11683 leader_election.cc:302] T e602722e9856465fbc3769d7752deaca P 1f265f2b8c5a422ba0c3e0f4d677918c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f265f2b8c5a422ba0c3e0f4d677918c, 4424247be5514d9b98cbdbb48b861d46; no voters:
I20260925 22:44:24.703161 12086 raft_consensus.cc:2781] T e602722e9856465fbc3769d7752deaca P 1f265f2b8c5a422ba0c3e0f4d677918c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 22:44:24.703244 12086 raft_consensus.cc:482] T e602722e9856465fbc3769d7752deaca P 1f265f2b8c5a422ba0c3e0f4d677918c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 22:44:24.703274 12086 raft_consensus.cc:3037] T e602722e9856465fbc3769d7752deaca P 1f265f2b8c5a422ba0c3e0f4d677918c [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:44:24.704491 12086 raft_consensus.cc:504] T e602722e9856465fbc3769d7752deaca P 1f265f2b8c5a422ba0c3e0f4d677918c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "276843c81a0e4c7c998527f76b02145d" member_type: VOTER last_known_addr { host: "127.10.198.2" port: 38273 } } peers { permanent_uuid: "1f265f2b8c5a422ba0c3e0f4d677918c" member_type: VOTER last_known_addr { host: "127.10.198.1" port: 38669 } } peers { permanent_uuid: "4424247be5514d9b98cbdbb48b861d46" member_type: VOTER last_known_addr { host: "127.10.198.3" port: 32859 } }
I20260925 22:44:24.704902 12086 leader_election.cc:288] T e602722e9856465fbc3769d7752deaca P 1f265f2b8c5a422ba0c3e0f4d677918c [CANDIDATE]: Term 1 election: Requested vote from peers 276843c81a0e4c7c998527f76b02145d (127.10.198.2:38273), 4424247be5514d9b98cbdbb48b861d46 (127.10.198.3:32859)
I20260925 22:44:24.705422 11881 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e602722e9856465fbc3769d7752deaca" candidate_uuid: "1f265f2b8c5a422ba0c3e0f4d677918c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "276843c81a0e4c7c998527f76b02145d"
I20260925 22:44:24.705574 11881 raft_consensus.cc:3037] T e602722e9856465fbc3769d7752deaca P 276843c81a0e4c7c998527f76b02145d [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:44:24.705693 12015 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e602722e9856465fbc3769d7752deaca" candidate_uuid: "1f265f2b8c5a422ba0c3e0f4d677918c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4424247be5514d9b98cbdbb48b861d46"
I20260925 22:44:24.705927 12015 raft_consensus.cc:3037] T e602722e9856465fbc3769d7752deaca P 4424247be5514d9b98cbdbb48b861d46 [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:44:24.706944 11881 raft_consensus.cc:2454] T e602722e9856465fbc3769d7752deaca P 276843c81a0e4c7c998527f76b02145d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1f265f2b8c5a422ba0c3e0f4d677918c in term 1.
I20260925 22:44:24.707145 12015 raft_consensus.cc:2454] T e602722e9856465fbc3769d7752deaca P 4424247be5514d9b98cbdbb48b861d46 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1f265f2b8c5a422ba0c3e0f4d677918c in term 1.
I20260925 22:44:24.707387 11682 leader_election.cc:302] T e602722e9856465fbc3769d7752deaca P 1f265f2b8c5a422ba0c3e0f4d677918c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f265f2b8c5a422ba0c3e0f4d677918c, 276843c81a0e4c7c998527f76b02145d; no voters:
I20260925 22:44:24.707589 12086 raft_consensus.cc:2781] T e602722e9856465fbc3769d7752deaca P 1f265f2b8c5a422ba0c3e0f4d677918c [term 1 FOLLOWER]: Leader election won for term 1
I20260925 22:44:24.707898 12086 raft_consensus.cc:686] T e602722e9856465fbc3769d7752deaca P 1f265f2b8c5a422ba0c3e0f4d677918c [term 1 LEADER]: Becoming Leader. State: Replica: 1f265f2b8c5a422ba0c3e0f4d677918c, State: Running, Role: LEADER
I20260925 22:44:24.708276 12086 consensus_queue.cc:237] T e602722e9856465fbc3769d7752deaca P 1f265f2b8c5a422ba0c3e0f4d677918c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "276843c81a0e4c7c998527f76b02145d" member_type: VOTER last_known_addr { host: "127.10.198.2" port: 38273 } } peers { permanent_uuid: "1f265f2b8c5a422ba0c3e0f4d677918c" member_type: VOTER last_known_addr { host: "127.10.198.1" port: 38669 } } peers { permanent_uuid: "4424247be5514d9b98cbdbb48b861d46" member_type: VOTER last_known_addr { host: "127.10.198.3" port: 32859 } }
I20260925 22:44:24.712080 11601 catalog_manager.cc:5495] T e602722e9856465fbc3769d7752deaca P 1f265f2b8c5a422ba0c3e0f4d677918c reported cstate change: term changed from 0 to 1, leader changed from <none> to 1f265f2b8c5a422ba0c3e0f4d677918c (127.10.198.1). New cstate: current_term: 1 leader_uuid: "1f265f2b8c5a422ba0c3e0f4d677918c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "276843c81a0e4c7c998527f76b02145d" member_type: VOTER last_known_addr { host: "127.10.198.2" port: 38273 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1f265f2b8c5a422ba0c3e0f4d677918c" member_type: VOTER last_known_addr { host: "127.10.198.1" port: 38669 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4424247be5514d9b98cbdbb48b861d46" member_type: VOTER last_known_addr { host: "127.10.198.3" port: 32859 } health_report { overall_health: UNKNOWN } } }
Sep 25 22:44:24 dist-test-slave-5vdr krb5kdc[11557](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376263, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.198.1@KRBTEST.COM
I20260925 22:44:24.731495 12076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:44:24.721723 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40452 (local address 127.10.198.1:38669)
0925 22:44:24.721899 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0925 22:44:24.721903 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:44:24.722060 (+ 157us) server_negotiation.cc:407] Connection header received
0925 22:44:24.722172 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:44:24.722175 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:44:24.722222 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:44:24.722297 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:44:24.722974 (+ 677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:44:24.723481 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:44:24.724469 (+ 988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:44:24.724685 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:44:24.724772 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:44:24.727755 (+ 2983us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:44:24.727768 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:44:24.727784 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:44:24.727825 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:44:24.729558 (+ 1733us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:44:24.730045 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:44:24.730050 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:44:24.730052 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:44:24.730159 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:44:24.730437 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:44:24.730441 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:44:24.730442 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:44:24.730748 (+ 306us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:44:24.730963 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0925 22:44:24.731130 (+ 167us) server_negotiation.cc:299] Negotiation successful
0925 22:44:24.731289 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
Sep 25 22:44:24 dist-test-slave-5vdr krb5kdc[11557](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376264, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 25 22:44:24 dist-test-slave-5vdr krb5kdc[11557](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376264, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.10.198.1@KRBTEST.COM
I20260925 22:44:24.761739 12076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:44:24.752072 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40460 (local address 127.10.198.1:38669)
0925 22:44:24.752250 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0925 22:44:24.752256 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 22:44:24.752479 (+ 223us) server_negotiation.cc:407] Connection header received
0925 22:44:24.752682 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:44:24.752686 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:44:24.752732 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:44:24.752826 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:44:24.753600 (+ 774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:44:24.754129 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:44:24.754992 (+ 863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:44:24.755223 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:44:24.755342 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:44:24.758180 (+ 2838us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:44:24.758184 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:44:24.758186 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:44:24.758219 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:44:24.759745 (+ 1526us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:44:24.760294 (+ 549us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:44:24.760298 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:44:24.760299 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:44:24.760391 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:44:24.760710 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:44:24.760713 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:44:24.760715 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:44:24.760906 (+ 191us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:44:24.761085 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0925 22:44:24.761439 (+ 354us) server_negotiation.cc:299] Negotiation successful
0925 22:44:24.761567 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
Sep 25 22:44:24 dist-test-slave-5vdr krb5kdc[11557](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376264, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 25 22:44:24 dist-test-slave-5vdr krb5kdc[11557](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376264, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.198.1@KRBTEST.COM
I20260925 22:44:24.790731 12076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:44:24.780720 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40466 (local address 127.10.198.1:38669)
0925 22:44:24.780914 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0925 22:44:24.780920 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 22:44:24.781098 (+ 178us) server_negotiation.cc:407] Connection header received
0925 22:44:24.781282 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:44:24.781288 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:44:24.781344 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:44:24.781449 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:44:24.782293 (+ 844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:44:24.782828 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:44:24.783791 (+ 963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:44:24.784089 (+ 298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:44:24.784220 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:44:24.787088 (+ 2868us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:44:24.787092 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:44:24.787095 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:44:24.787131 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:44:24.788820 (+ 1689us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:44:24.789381 (+ 561us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:44:24.789384 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:44:24.789386 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:44:24.789498 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:44:24.789762 (+ 264us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:44:24.789764 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:44:24.789765 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:44:24.789955 (+ 190us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:44:24.790091 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0925 22:44:24.790385 (+ 294us) server_negotiation.cc:299] Negotiation successful
0925 22:44:24.790525 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":99}
W20260925 22:44:24.792280 11707 tablet_service.cc:2908] Rejecting scan request for tablet e602722e9856465fbc3769d7752deaca: Uninitialized: clean time has not yet been initialized
I20260925 22:44:24.794500 11032 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrWXm8X/build/debug/bin/kudu with pid 11664
I20260925 22:44:24.800854 11032 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrWXm8X/build/debug/bin/kudu with pid 11798
I20260925 22:44:24.806572 11032 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrWXm8X/build/debug/bin/kudu with pid 11932
I20260925 22:44:24.812752 11032 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrWXm8X/build/debug/bin/kudu with pid 11570
2026-09-25T22:44:24Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (2428 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790376259650642-11032-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 22:44:24 dist-test-slave-5vdr krb5kdc[12103](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 22:44:24 dist-test-slave-5vdr krb5kdc[12103](info): set up 2 sockets
Sep 25 22:44:24 dist-test-slave-5vdr krb5kdc[12103](info): commencing operation
krb5kdc: starting...
W20260925 22:44:25.859869 11032 mini_kdc.cc:121] Time spent starting KDC: real 1.023s user 0.004s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790376259650642-11032-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790376259650642-11032-0/krb5kdc/test-user.keytab.
Sep 25 22:44:25 dist-test-slave-5vdr krb5kdc[12103](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376265, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T22:44:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T22:44:25Z Disabled control of system clock
WARNING: no policy specified for kudu/127.10.198.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.198.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.198.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790376259650642-11032-0/krb5kdc/kudu_127.10.198.62.keytab.
Entry for principal kudu/127.10.198.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790376259650642-11032-0/krb5kdc/kudu_127.10.198.62.keytab.
I20260925 22:44:25.991518 11032 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrWXm8X/build/debug/bin/kudu
/tmp/dist-test-taskrWXm8X/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790376259650642-11032-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790376259650642-11032-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790376259650642-11032-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790376259650642-11032-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.198.62:42797
--webserver_interface=127.10.198.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790376259650642-11032-0/krb5kdc/kudu_127.10.198.62.keytab
--principal=kudu/127.10.198.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.198.20:37397
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.198.62:42797
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790376259650642-11032-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790376259650642-11032-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790376259650642-11032-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:44:26.113669 12116 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:44:26.114058 12116 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:44:26.114133 12116 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:44:26.118604 12116 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 22:44:26.118731 12116 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260925 22:44:26.118765 12116 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:44:26.118816 12116 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:44:26.118878 12116 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 22:44:26.118916 12116 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 22:44:26.124912 12116 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.198.20:37397
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790376259650642-11032-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790376259650642-11032-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.198.62:42797
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790376259650642-11032-0/krb5kdc/kudu_127.10.198.62.keytab
--principal=kudu/127.10.198.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.198.62:42797
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790376259650642-11032-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.198.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12116
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790376259650642-11032-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:09:16 UTC on ccycloud.quasar-afmhyw.root.comops.site
build id 4673379
I20260925 22:44:26.126181 12116 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:44:26.127352 12116 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:44:26.136693 12122 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:44:26.137107 12124 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:44:26.137123 12121 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:44:26.137392 12116 server_base.cc:1034] running on GCE node
I20260925 22:44:26.138226 12116 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:44:26.139637 12116 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:44:26.140902 12116 hybrid_clock.cc:648] HybridClock initialized: now 1790376266140871 us; error 57 us; skew 500 ppm
Sep 25 22:44:26 dist-test-slave-5vdr krb5kdc[12103](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376266, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.198.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:44:26.148129 12116 init.cc:377] Logged in from keytab as kudu/127.10.198.62@KRBTEST.COM (short username kudu)
I20260925 22:44:26.149307 12116 webserver.cc:460] Webserver started at http://127.10.198.62:35541/ using document root <none> and password file <none>
I20260925 22:44:26.149993 12116 fs_manager.cc:359] Metadata directory not provided
I20260925 22:44:26.150112 12116 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:44:26.150386 12116 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:44:26.152829 12116 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790376259650642-11032-0/minicluster-data/master-0/data/instance:
uuid: "d5451cd6c49743beb1c5b14b5163a164"
format_stamp: "Formatted at 2026-09-25 22:44:26 on dist-test-slave-5vdr"
server_key: "45f7372f8023fec1adfdd056c327bce8"
server_key_iv: "40e5bb08f960b14be0687ab2577f7534"
server_key_version: "encryptionkey@0"
I20260925 22:44:26.153487 12116 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790376259650642-11032-0/minicluster-data/master-0/wal/instance:
uuid: "d5451cd6c49743beb1c5b14b5163a164"
format_stamp: "Formatted at 2026-09-25 22:44:26 on dist-test-slave-5vdr"
server_key: "45f7372f8023fec1adfdd056c327bce8"
server_key_iv: "40e5bb08f960b14be0687ab2577f7534"
server_key_version: "encryptionkey@0"
I20260925 22:44:26.157943 12116 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260925 22:44:26.161019 12131 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:44:26.162593 12116 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.004s
I20260925 22:44:26.162786 12116 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790376259650642-11032-0/minicluster-data/master-0/data,/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790376259650642-11032-0/minicluster-data/master-0/wal
uuid: "d5451cd6c49743beb1c5b14b5163a164"
format_stamp: "Formatted at 2026-09-25 22:44:26 on dist-test-slave-5vdr"
server_key: "45f7372f8023fec1adfdd056c327bce8"
server_key_iv: "40e5bb08f960b14be0687ab2577f7534"
server_key_version: "encryptionkey@0"
I20260925 22:44:26.162949 12116 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790376259650642-11032-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790376259650642-11032-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790376259650642-11032-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:44:26.185518 12116 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:44:26.189625 12116 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:44:26.189873 12116 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:44:26.198885 12116 rpc_server.cc:307] RPC server started. Bound to: 127.10.198.62:42797
I20260925 22:44:26.198894 12193 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.198.62:42797 every 8 connection(s)
I20260925 22:44:26.200076 12116 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790376259650642-11032-0/minicluster-data/master-0/data/info.pb
I20260925 22:44:26.204025 12194 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:44:26.209337 11032 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrWXm8X/build/debug/bin/kudu as pid 12116
I20260925 22:44:26.209510 11032 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790376259650642-11032-0/minicluster-data/master-0/wal/instance
I20260925 22:44:26.209790 11032 external_mini_cluster.cc:1428] Setting key 6fdd1d05aa09d4eb87d7fa7ce90d96c2
I20260925 22:44:26.211645 12194 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d5451cd6c49743beb1c5b14b5163a164: Bootstrap starting.
I20260925 22:44:26.214550 12194 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d5451cd6c49743beb1c5b14b5163a164: Neither blocks nor log segments found. Creating new log.
I20260925 22:44:26.215533 12194 log.cc:824] T 00000000000000000000000000000000 P d5451cd6c49743beb1c5b14b5163a164: Log is configured to *not* fsync() on all Append() calls
I20260925 22:44:26.218402 12194 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d5451cd6c49743beb1c5b14b5163a164: No bootstrap required, opened a new log
Sep 25 22:44:26 dist-test-slave-5vdr krb5kdc[12103](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376265, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.198.62@KRBTEST.COM
I20260925 22:44:26.222997 12194 raft_consensus.cc:348] T 00000000000000000000000000000000 P d5451cd6c49743beb1c5b14b5163a164 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d5451cd6c49743beb1c5b14b5163a164" member_type: VOTER last_known_addr { host: "127.10.198.62" port: 42797 } }
I20260925 22:44:26.223351 12194 raft_consensus.cc:374] T 00000000000000000000000000000000 P d5451cd6c49743beb1c5b14b5163a164 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:44:26.223448 12194 raft_consensus.cc:729] T 00000000000000000000000000000000 P d5451cd6c49743beb1c5b14b5163a164 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d5451cd6c49743beb1c5b14b5163a164, State: Initialized, Role: FOLLOWER
I20260925 22:44:26.224076 12194 consensus_queue.cc:260] T 00000000000000000000000000000000 P d5451cd6c49743beb1c5b14b5163a164 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d5451cd6c49743beb1c5b14b5163a164" member_type: VOTER last_known_addr { host: "127.10.198.62" port: 42797 } }
I20260925 22:44:26.224272 12194 raft_consensus.cc:388] T 00000000000000000000000000000000 P d5451cd6c49743beb1c5b14b5163a164 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 22:44:26.224362 12194 raft_consensus.cc:482] T 00000000000000000000000000000000 P d5451cd6c49743beb1c5b14b5163a164 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 22:44:26.224516 12194 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d5451cd6c49743beb1c5b14b5163a164 [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:44:26.225631 12194 raft_consensus.cc:504] T 00000000000000000000000000000000 P d5451cd6c49743beb1c5b14b5163a164 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d5451cd6c49743beb1c5b14b5163a164" member_type: VOTER last_known_addr { host: "127.10.198.62" port: 42797 } }
I20260925 22:44:26.226150 12194 leader_election.cc:302] T 00000000000000000000000000000000 P d5451cd6c49743beb1c5b14b5163a164 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d5451cd6c49743beb1c5b14b5163a164; no voters:
I20260925 22:44:26.226589 12194 leader_election.cc:288] T 00000000000000000000000000000000 P d5451cd6c49743beb1c5b14b5163a164 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 22:44:26.226735 12199 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d5451cd6c49743beb1c5b14b5163a164 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 22:44:26.227008 12199 raft_consensus.cc:686] T 00000000000000000000000000000000 P d5451cd6c49743beb1c5b14b5163a164 [term 1 LEADER]: Becoming Leader. State: Replica: d5451cd6c49743beb1c5b14b5163a164, State: Running, Role: LEADER
I20260925 22:44:26.227324 12197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:44:26.211481 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49800 (local address 127.10.198.62:42797)
0925 22:44:26.212160 (+ 679us) server_negotiation.cc:206] Beginning negotiation
0925 22:44:26.212180 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0925 22:44:26.212231 (+ 51us) server_negotiation.cc:407] Connection header received
0925 22:44:26.212991 (+ 760us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:44:26.213012 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:44:26.213435 (+ 423us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:44:26.213867 (+ 432us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:44:26.214779 (+ 912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:44:26.216002 (+ 1223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:44:26.217360 (+ 1358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:44:26.217895 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:44:26.218030 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:44:26.221322 (+ 3292us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:44:26.221328 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:44:26.221346 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:44:26.221383 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:44:26.223433 (+ 2050us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:44:26.224097 (+ 664us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:44:26.224108 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:44:26.224113 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:44:26.224267 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:44:26.224616 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:44:26.224620 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:44:26.224622 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:44:26.225084 (+ 462us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:44:26.225496 (+ 412us) server_negotiation.cc:1035] Waiting for connection context
0925 22:44:26.225845 (+ 349us) server_negotiation.cc:299] Negotiation successful
0925 22:44:26.226216 (+ 371us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":495,"thread_start_us":177,"threads_started":1}
I20260925 22:44:26.227514 12199 consensus_queue.cc:237] T 00000000000000000000000000000000 P d5451cd6c49743beb1c5b14b5163a164 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d5451cd6c49743beb1c5b14b5163a164" member_type: VOTER last_known_addr { host: "127.10.198.62" port: 42797 } }
I20260925 22:44:26.228248 12194 sys_catalog.cc:564] T 00000000000000000000000000000000 P d5451cd6c49743beb1c5b14b5163a164 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 22:44:26.229631 12200 sys_catalog.cc:455] T 00000000000000000000000000000000 P d5451cd6c49743beb1c5b14b5163a164 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d5451cd6c49743beb1c5b14b5163a164" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d5451cd6c49743beb1c5b14b5163a164" member_type: VOTER last_known_addr { host: "127.10.198.62" port: 42797 } } }
I20260925 22:44:26.229799 12200 sys_catalog.cc:458] T 00000000000000000000000000000000 P d5451cd6c49743beb1c5b14b5163a164 [sys.catalog]: This master's current role is: LEADER
I20260925 22:44:26.230159 12201 sys_catalog.cc:455] T 00000000000000000000000000000000 P d5451cd6c49743beb1c5b14b5163a164 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d5451cd6c49743beb1c5b14b5163a164. Latest consensus state: current_term: 1 leader_uuid: "d5451cd6c49743beb1c5b14b5163a164" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d5451cd6c49743beb1c5b14b5163a164" member_type: VOTER last_known_addr { host: "127.10.198.62" port: 42797 } } }
I20260925 22:44:26.230269 12201 sys_catalog.cc:458] T 00000000000000000000000000000000 P d5451cd6c49743beb1c5b14b5163a164 [sys.catalog]: This master's current role is: LEADER
W20260925 22:44:26.234004 12214 catalog_manager.cc:1558] T 00000000000000000000000000000000 P d5451cd6c49743beb1c5b14b5163a164: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 22:44:26.234114 12214 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 22:44:26.234229 12208 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 22:44:26.235090 12208 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 22:44:26.242412 12208 catalog_manager.cc:1347] Generated new cluster ID: 1a144a4aa721489f8f708a622fbc7367
I20260925 22:44:26.242543 12208 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 22:44:26.262341 12208 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 22:44:26.263424 12208 catalog_manager.cc:1504] Loading token signing keys...
I20260925 22:44:26.268065 12208 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d5451cd6c49743beb1c5b14b5163a164: Generated new TSK 0
I20260925 22:44:26.269047 12208 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.10.198.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.198.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.198.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790376259650642-11032-0/krb5kdc/kudu_127.10.198.1.keytab.
Entry for principal kudu/127.10.198.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790376259650642-11032-0/krb5kdc/kudu_127.10.198.1.keytab.
I20260925 22:44:26.303437 11032 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrWXm8X/build/debug/bin/kudu
/tmp/dist-test-taskrWXm8X/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790376259650642-11032-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790376259650642-11032-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790376259650642-11032-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790376259650642-11032-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.198.1:0
--local_ip_for_outbound_sockets=127.10.198.1
--webserver_interface=127.10.198.1
--webserver_port=0
--tserver_master_addrs=127.10.198.62:42797
--keytab_file=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790376259650642-11032-0/krb5kdc/kudu_127.10.198.1.keytab
--principal=kudu/127.10.198.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.198.20:37397
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790376259650642-11032-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790376259650642-11032-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790376259650642-11032-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:44:26.426654 12220 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:44:26.427053 12220 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:44:26.427150 12220 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:44:26.431833 12220 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:44:26.431998 12220 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:44:26.432072 12220 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260925 22:44:26.432161 12220 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.198.1
I20260925 22:44:26.438164 12220 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.198.20:37397
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790376259650642-11032-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790376259650642-11032-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790376259650642-11032-0/krb5kdc/kudu_127.10.198.1.keytab
--principal=kudu/127.10.198.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.198.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790376259650642-11032-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.198.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.10.198.62:42797
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12220
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.198.1
--log_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790376259650642-11032-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:09:16 UTC on ccycloud.quasar-afmhyw.root.comops.site
build id 4673379
I20260925 22:44:26.439402 12220 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:44:26.440577 12220 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:44:26.451421 12228 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:44:26.451411 12225 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:44:26.451537 12220 server_base.cc:1034] running on GCE node
W20260925 22:44:26.451411 12226 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:44:26.452296 12220 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:44:26.453095 12220 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:44:26.454334 12220 hybrid_clock.cc:648] HybridClock initialized: now 1790376266454315 us; error 39 us; skew 500 ppm
Sep 25 22:44:26 dist-test-slave-5vdr krb5kdc[12103](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376266, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.198.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:44:26.457536 12220 init.cc:377] Logged in from keytab as kudu/127.10.198.1@KRBTEST.COM (short username kudu)
I20260925 22:44:26.458598 12220 webserver.cc:460] Webserver started at http://127.10.198.1:46003/ using document root <none> and password file <none>
I20260925 22:44:26.459249 12220 fs_manager.cc:359] Metadata directory not provided
I20260925 22:44:26.459355 12220 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:44:26.459631 12220 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:44:26.462073 12220 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790376259650642-11032-0/minicluster-data/ts-0/data/instance:
uuid: "3065b552ce554279b34368a4fec28dd2"
format_stamp: "Formatted at 2026-09-25 22:44:26 on dist-test-slave-5vdr"
server_key: "ff933b8d26097f37541d80b8f0d2b936"
server_key_iv: "54dc63f6d01279cc5b6dbd75c4a7cfd9"
server_key_version: "encryptionkey@0"
I20260925 22:44:26.462730 12220 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790376259650642-11032-0/minicluster-data/ts-0/wal/instance:
uuid: "3065b552ce554279b34368a4fec28dd2"
format_stamp: "Formatted at 2026-09-25 22:44:26 on dist-test-slave-5vdr"
server_key: "ff933b8d26097f37541d80b8f0d2b936"
server_key_iv: "54dc63f6d01279cc5b6dbd75c4a7cfd9"
server_key_version: "encryptionkey@0"
I20260925 22:44:26.467237 12220 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260925 22:44:26.470182 12235 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:44:26.471729 12220 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260925 22:44:26.471947 12220 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790376259650642-11032-0/minicluster-data/ts-0/data,/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790376259650642-11032-0/minicluster-data/ts-0/wal
uuid: "3065b552ce554279b34368a4fec28dd2"
format_stamp: "Formatted at 2026-09-25 22:44:26 on dist-test-slave-5vdr"
server_key: "ff933b8d26097f37541d80b8f0d2b936"
server_key_iv: "54dc63f6d01279cc5b6dbd75c4a7cfd9"
server_key_version: "encryptionkey@0"
I20260925 22:44:26.472097 12220 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790376259650642-11032-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790376259650642-11032-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790376259650642-11032-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:44:26.489086 12220 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:44:26.492908 12220 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:44:26.493161 12220 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:44:26.494802 12220 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:44:26.494895 12220 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:44:26.494977 12220 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:44:26.495023 12220 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 25 22:44:26 dist-test-slave-5vdr krb5kdc[12103](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376266, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.198.1@KRBTEST.COM for kudu/127.10.198.62@KRBTEST.COM
I20260925 22:44:26.507213 12220 rpc_server.cc:307] RPC server started. Bound to: 127.10.198.1:38019
I20260925 22:44:26.507297 12350 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.198.1:38019 every 8 connection(s)
I20260925 22:44:26.508496 12220 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790376259650642-11032-0/minicluster-data/ts-0/data/info.pb
I20260925 22:44:26.511590 11032 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrWXm8X/build/debug/bin/kudu as pid 12220
I20260925 22:44:26.511888 11032 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790376259650642-11032-0/minicluster-data/ts-0/wal/instance
I20260925 22:44:26.512269 11032 external_mini_cluster.cc:1428] Setting key d5b911a70c23551d7e37aa92daf8931c
I20260925 22:44:26.512413 12197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:44:26.496122 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.198.1:47703 (local address 127.10.198.62:42797)
0925 22:44:26.496274 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0925 22:44:26.496278 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:44:26.497698 (+ 1420us) server_negotiation.cc:407] Connection header received
0925 22:44:26.499248 (+ 1550us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:44:26.499256 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:44:26.499344 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:44:26.499465 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:44:26.501282 (+ 1817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:44:26.502235 (+ 953us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:44:26.503965 (+ 1730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:44:26.504248 (+ 283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:44:26.504386 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:44:26.507691 (+ 3305us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:44:26.507694 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:44:26.507696 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:44:26.507727 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:44:26.509603 (+ 1876us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:44:26.510179 (+ 576us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:44:26.510184 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:44:26.510185 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:44:26.510290 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:44:26.510796 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:44:26.510800 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:44:26.510801 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:44:26.511010 (+ 209us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:44:26.511101 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0925 22:44:26.512047 (+ 946us) server_negotiation.cc:299] Negotiation successful
0925 22:44:26.512221 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260925 22:44:26.513382 12244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:44:26.496808 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.198.62:42797 (local address 127.10.198.1:47703)
0925 22:44:26.497471 (+ 663us) negotiation.cc:107] Waiting for socket to connect
0925 22:44:26.497524 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0925 22:44:26.498862 (+ 1338us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:44:26.499675 (+ 813us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:44:26.499683 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:44:26.500290 (+ 607us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:44:26.500981 (+ 691us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:44:26.500998 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:44:26.502449 (+ 1451us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:44:26.502460 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:44:26.503743 (+ 1283us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:44:26.503757 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:44:26.504436 (+ 679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:44:26.504441 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:44:26.504652 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:44:26.505502 (+ 850us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:44:26.505526 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:44:26.507433 (+ 1907us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:44:26.509735 (+ 2302us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:44:26.509741 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:44:26.509755 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:44:26.510047 (+ 292us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:44:26.510431 (+ 384us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:44:26.510435 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:44:26.510436 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:44:26.510603 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:44:26.511248 (+ 645us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:44:26.511254 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:44:26.511680 (+ 426us) client_negotiation.cc:769] Sending connection context
0925 22:44:26.511961 (+ 281us) client_negotiation.cc:240] Negotiation successful
0925 22:44:26.512372 (+ 411us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":432,"thread_start_us":173,"threads_started":1}
I20260925 22:44:26.515223 12351 heartbeater.cc:347] Connected to a master server at 127.10.198.62:42797
I20260925 22:44:26.515408 12351 heartbeater.cc:464] Registering TS with master...
I20260925 22:44:26.516155 12351 heartbeater.cc:511] Master 127.10.198.62:42797 requested a full tablet report, sending...
I20260925 22:44:26.518191 12147 ts_manager.cc:194] Registered new tserver with Master: 3065b552ce554279b34368a4fec28dd2 (127.10.198.1:38019)
I20260925 22:44:26.520107 12147 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.198.1@KRBTEST.COM'} at 127.10.198.1:47703
WARNING: no policy specified for kudu/127.10.198.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.198.2@KRBTEST.COM" created.
I20260925 22:44:26.528788 12197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:44:26.520414 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.198.1:52991 (local address 127.10.198.62:42797)
0925 22:44:26.520562 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0925 22:44:26.520568 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 22:44:26.520960 (+ 392us) server_negotiation.cc:407] Connection header received
0925 22:44:26.521140 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:44:26.521145 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:44:26.521207 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:44:26.521308 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:44:26.521920 (+ 612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:44:26.522457 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:44:26.523368 (+ 911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:44:26.523593 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:44:26.523655 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:44:26.525032 (+ 1377us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:44:26.525037 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:44:26.525045 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:44:26.525073 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:44:26.526859 (+ 1786us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:44:26.527357 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:44:26.527361 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:44:26.527363 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:44:26.527457 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:44:26.527779 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:44:26.527783 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:44:26.527785 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:44:26.528065 (+ 280us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:44:26.528181 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0925 22:44:26.528467 (+ 286us) server_negotiation.cc:299] Negotiation successful
0925 22:44:26.528584 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260925 22:44:26.528796 12358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:44:26.520519 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.198.62:42797 (local address 127.10.198.1:52991)
0925 22:44:26.520858 (+ 339us) negotiation.cc:107] Waiting for socket to connect
0925 22:44:26.520871 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 22:44:26.521018 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:44:26.521348 (+ 330us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:44:26.521351 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:44:26.521592 (+ 241us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:44:26.521778 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:44:26.521785 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:44:26.522598 (+ 813us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:44:26.522602 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:44:26.523186 (+ 584us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:44:26.523194 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:44:26.523811 (+ 617us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:44:26.523813 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:44:26.523952 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:44:26.524480 (+ 528us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:44:26.524494 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:44:26.524842 (+ 348us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:44:26.527001 (+ 2159us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:44:26.527005 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:44:26.527007 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:44:26.527244 (+ 237us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:44:26.527560 (+ 316us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:44:26.527564 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:44:26.527566 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:44:26.527663 (+ 97us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:44:26.528174 (+ 511us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:44:26.528178 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:44:26.528291 (+ 113us) client_negotiation.cc:769] Sending connection context
0925 22:44:26.528407 (+ 116us) client_negotiation.cc:240] Negotiation successful
0925 22:44:26.528585 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":234,"mutex_wait_us":3,"thread_start_us":140,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.198.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790376259650642-11032-0/krb5kdc/kudu_127.10.198.2.keytab.
Entry for principal kudu/127.10.198.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790376259650642-11032-0/krb5kdc/kudu_127.10.198.2.keytab.
I20260925 22:44:26.546677 11032 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrWXm8X/build/debug/bin/kudu
/tmp/dist-test-taskrWXm8X/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790376259650642-11032-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790376259650642-11032-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790376259650642-11032-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790376259650642-11032-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.198.2:0
--local_ip_for_outbound_sockets=127.10.198.2
--webserver_interface=127.10.198.2
--webserver_port=0
--tserver_master_addrs=127.10.198.62:42797
--keytab_file=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790376259650642-11032-0/krb5kdc/kudu_127.10.198.2.keytab
--principal=kudu/127.10.198.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.198.20:37397
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790376259650642-11032-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790376259650642-11032-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790376259650642-11032-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:44:26.671981 12360 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:44:26.672297 12360 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:44:26.672344 12360 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:44:26.676849 12360 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:44:26.676932 12360 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:44:26.676975 12360 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260925 22:44:26.677066 12360 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.198.2
I20260925 22:44:26.682907 12360 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.198.20:37397
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790376259650642-11032-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790376259650642-11032-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790376259650642-11032-0/krb5kdc/kudu_127.10.198.2.keytab
--principal=kudu/127.10.198.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.198.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790376259650642-11032-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.198.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.10.198.62:42797
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12360
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.198.2
--log_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790376259650642-11032-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:09:16 UTC on ccycloud.quasar-afmhyw.root.comops.site
build id 4673379
I20260925 22:44:26.684158 12360 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:44:26.685343 12360 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:44:26.696602 12365 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:44:26.696602 12366 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:44:26.696657 12368 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:44:26.697098 12360 server_base.cc:1034] running on GCE node
I20260925 22:44:26.697608 12360 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:44:26.698385 12360 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:44:26.699595 12360 hybrid_clock.cc:648] HybridClock initialized: now 1790376266699558 us; error 56 us; skew 500 ppm
Sep 25 22:44:26 dist-test-slave-5vdr krb5kdc[12103](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376266, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.198.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:44:26.702747 12360 init.cc:377] Logged in from keytab as kudu/127.10.198.2@KRBTEST.COM (short username kudu)
I20260925 22:44:26.703945 12360 webserver.cc:460] Webserver started at http://127.10.198.2:41837/ using document root <none> and password file <none>
I20260925 22:44:26.704599 12360 fs_manager.cc:359] Metadata directory not provided
I20260925 22:44:26.704705 12360 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:44:26.704963 12360 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:44:26.707232 12360 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790376259650642-11032-0/minicluster-data/ts-1/data/instance:
uuid: "e58bbdc59c354af99870745d454b2612"
format_stamp: "Formatted at 2026-09-25 22:44:26 on dist-test-slave-5vdr"
server_key: "e0744bdcc673664d222756aee7793cf4"
server_key_iv: "994ec7d34358c05f2dfa8eba1fd25615"
server_key_version: "encryptionkey@0"
I20260925 22:44:26.708020 12360 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790376259650642-11032-0/minicluster-data/ts-1/wal/instance:
uuid: "e58bbdc59c354af99870745d454b2612"
format_stamp: "Formatted at 2026-09-25 22:44:26 on dist-test-slave-5vdr"
server_key: "e0744bdcc673664d222756aee7793cf4"
server_key_iv: "994ec7d34358c05f2dfa8eba1fd25615"
server_key_version: "encryptionkey@0"
I20260925 22:44:26.712579 12360 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260925 22:44:26.715694 12375 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:44:26.717339 12360 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 22:44:26.717536 12360 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790376259650642-11032-0/minicluster-data/ts-1/data,/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790376259650642-11032-0/minicluster-data/ts-1/wal
uuid: "e58bbdc59c354af99870745d454b2612"
format_stamp: "Formatted at 2026-09-25 22:44:26 on dist-test-slave-5vdr"
server_key: "e0744bdcc673664d222756aee7793cf4"
server_key_iv: "994ec7d34358c05f2dfa8eba1fd25615"
server_key_version: "encryptionkey@0"
I20260925 22:44:26.717687 12360 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790376259650642-11032-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790376259650642-11032-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790376259650642-11032-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:44:26.731101 12360 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:44:26.734894 12360 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:44:26.735128 12360 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:44:26.736863 12360 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:44:26.736931 12360 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:44:26.736974 12360 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:44:26.736990 12360 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 22:44:26.748682 12360 rpc_server.cc:307] RPC server started. Bound to: 127.10.198.2:34917
I20260925 22:44:26.748777 12490 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.198.2:34917 every 8 connection(s)
Sep 25 22:44:26 dist-test-slave-5vdr krb5kdc[12103](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376266, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.198.2@KRBTEST.COM for kudu/127.10.198.62@KRBTEST.COM
I20260925 22:44:26.749900 12360 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790376259650642-11032-0/minicluster-data/ts-1/data/info.pb
I20260925 22:44:26.753991 11032 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrWXm8X/build/debug/bin/kudu as pid 12360
I20260925 22:44:26.754134 11032 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790376259650642-11032-0/minicluster-data/ts-1/wal/instance
I20260925 22:44:26.754402 11032 external_mini_cluster.cc:1428] Setting key ca5e61f6ec594c67080d7c84cd5316de
I20260925 22:44:26.754537 12197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:44:26.738778 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.198.2:34751 (local address 127.10.198.62:42797)
0925 22:44:26.738972 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0925 22:44:26.738978 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 22:44:26.740229 (+ 1251us) server_negotiation.cc:407] Connection header received
0925 22:44:26.741776 (+ 1547us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:44:26.741780 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:44:26.741857 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:44:26.741978 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:44:26.744122 (+ 2144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:44:26.745041 (+ 919us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:44:26.746531 (+ 1490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:44:26.746935 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:44:26.747069 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:44:26.750600 (+ 3531us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:44:26.750604 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:44:26.750607 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:44:26.750643 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:44:26.752191 (+ 1548us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:44:26.752832 (+ 641us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:44:26.752835 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:44:26.752836 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:44:26.752922 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:44:26.753282 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:44:26.753284 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:44:26.753285 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:44:26.753535 (+ 250us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:44:26.753621 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0925 22:44:26.754225 (+ 604us) server_negotiation.cc:299] Negotiation successful
0925 22:44:26.754367 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260925 22:44:26.755590 12386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:44:26.739304 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.198.62:42797 (local address 127.10.198.2:34751)
0925 22:44:26.739997 (+ 693us) negotiation.cc:107] Waiting for socket to connect
0925 22:44:26.740073 (+ 76us) client_negotiation.cc:174] Beginning negotiation
0925 22:44:26.741467 (+ 1394us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:44:26.742254 (+ 787us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:44:26.742270 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:44:26.742907 (+ 637us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:44:26.743888 (+ 981us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:44:26.743919 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:44:26.745268 (+ 1349us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:44:26.745271 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:44:26.746370 (+ 1099us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:44:26.746388 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:44:26.747187 (+ 799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:44:26.747197 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:44:26.747454 (+ 257us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:44:26.748302 (+ 848us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:44:26.748331 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:44:26.750312 (+ 1981us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:44:26.752340 (+ 2028us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:44:26.752346 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:44:26.752361 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:44:26.752710 (+ 349us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:44:26.753007 (+ 297us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:44:26.753009 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:44:26.753011 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:44:26.753156 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:44:26.753662 (+ 506us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:44:26.753668 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:44:26.753941 (+ 273us) client_negotiation.cc:769] Sending connection context
0925 22:44:26.754166 (+ 225us) client_negotiation.cc:240] Negotiation successful
0925 22:44:26.754533 (+ 367us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":364,"thread_start_us":168,"threads_started":1}
I20260925 22:44:26.757958 12491 heartbeater.cc:347] Connected to a master server at 127.10.198.62:42797
I20260925 22:44:26.758175 12491 heartbeater.cc:464] Registering TS with master...
I20260925 22:44:26.758821 12491 heartbeater.cc:511] Master 127.10.198.62:42797 requested a full tablet report, sending...
I20260925 22:44:26.760142 12148 ts_manager.cc:194] Registered new tserver with Master: e58bbdc59c354af99870745d454b2612 (127.10.198.2:34917)
I20260925 22:44:26.760996 12148 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.198.2@KRBTEST.COM'} at 127.10.198.2:34751
WARNING: no policy specified for kudu/127.10.198.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.198.3@KRBTEST.COM" created.
I20260925 22:44:26.769362 12498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:44:26.759724 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.198.62:42797 (local address 127.10.198.2:33167)
0925 22:44:26.760294 (+ 570us) negotiation.cc:107] Waiting for socket to connect
0925 22:44:26.760310 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 22:44:26.760435 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:44:26.760744 (+ 309us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:44:26.760747 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:44:26.760989 (+ 242us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:44:26.761137 (+ 148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:44:26.761145 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:44:26.762460 (+ 1315us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:44:26.762463 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:44:26.763270 (+ 807us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:44:26.763282 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:44:26.764029 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:44:26.764032 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:44:26.764122 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:44:26.764744 (+ 622us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:44:26.764765 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:44:26.765119 (+ 354us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:44:26.767189 (+ 2070us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:44:26.767193 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:44:26.767195 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:44:26.767498 (+ 303us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:44:26.767940 (+ 442us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:44:26.767943 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:44:26.767945 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:44:26.768050 (+ 105us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:44:26.768828 (+ 778us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:44:26.768830 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:44:26.768933 (+ 103us) client_negotiation.cc:769] Sending connection context
0925 22:44:26.769039 (+ 106us) client_negotiation.cc:240] Negotiation successful
0925 22:44:26.769184 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":468,"thread_start_us":137,"threads_started":1}
I20260925 22:44:26.769722 12197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:44:26.760062 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.198.2:33167 (local address 127.10.198.62:42797)
0925 22:44:26.760240 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0925 22:44:26.760247 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 22:44:26.760410 (+ 163us) server_negotiation.cc:407] Connection header received
0925 22:44:26.760523 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:44:26.760529 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:44:26.760593 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:44:26.760699 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:44:26.761277 (+ 578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:44:26.762272 (+ 995us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:44:26.763441 (+ 1169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:44:26.763892 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:44:26.764028 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:44:26.765312 (+ 1284us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:44:26.765316 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:44:26.765328 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:44:26.765359 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:44:26.767045 (+ 1686us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:44:26.767637 (+ 592us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:44:26.767646 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:44:26.767648 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:44:26.767801 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:44:26.768179 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:44:26.768184 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:44:26.768186 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:44:26.768685 (+ 499us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:44:26.768827 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0925 22:44:26.769137 (+ 310us) server_negotiation.cc:299] Negotiation successful
0925 22:44:26.769302 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.198.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790376259650642-11032-0/krb5kdc/kudu_127.10.198.3.keytab.
Entry for principal kudu/127.10.198.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790376259650642-11032-0/krb5kdc/kudu_127.10.198.3.keytab.
I20260925 22:44:26.788278 11032 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrWXm8X/build/debug/bin/kudu
/tmp/dist-test-taskrWXm8X/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790376259650642-11032-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790376259650642-11032-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790376259650642-11032-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790376259650642-11032-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.198.3:0
--local_ip_for_outbound_sockets=127.10.198.3
--webserver_interface=127.10.198.3
--webserver_port=0
--tserver_master_addrs=127.10.198.62:42797
--keytab_file=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790376259650642-11032-0/krb5kdc/kudu_127.10.198.3.keytab
--principal=kudu/127.10.198.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.198.20:37397
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790376259650642-11032-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790376259650642-11032-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790376259650642-11032-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:44:26.912534 12500 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:44:26.912845 12500 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:44:26.912894 12500 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:44:26.917469 12500 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:44:26.917555 12500 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:44:26.917599 12500 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260925 22:44:26.917634 12500 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.198.3
I20260925 22:44:26.923578 12500 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.198.20:37397
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790376259650642-11032-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790376259650642-11032-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790376259650642-11032-0/krb5kdc/kudu_127.10.198.3.keytab
--principal=kudu/127.10.198.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.198.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790376259650642-11032-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.198.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.10.198.62:42797
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12500
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.198.3
--log_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790376259650642-11032-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:09:16 UTC on ccycloud.quasar-afmhyw.root.comops.site
build id 4673379
I20260925 22:44:26.924876 12500 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:44:26.926360 12500 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 22:44:26.937307 12500 server_base.cc:1034] running on GCE node
W20260925 22:44:26.937243 12508 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:44:26.937243 12506 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:44:26.937238 12505 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:44:26.938151 12500 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:44:26.938880 12500 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:44:26.940056 12500 hybrid_clock.cc:648] HybridClock initialized: now 1790376266940027 us; error 57 us; skew 500 ppm
Sep 25 22:44:26 dist-test-slave-5vdr krb5kdc[12103](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376266, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.198.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:44:26.943250 12500 init.cc:377] Logged in from keytab as kudu/127.10.198.3@KRBTEST.COM (short username kudu)
I20260925 22:44:26.944403 12500 webserver.cc:460] Webserver started at http://127.10.198.3:45777/ using document root <none> and password file <none>
I20260925 22:44:26.945052 12500 fs_manager.cc:359] Metadata directory not provided
I20260925 22:44:26.945164 12500 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:44:26.945420 12500 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:44:26.947793 12500 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790376259650642-11032-0/minicluster-data/ts-2/data/instance:
uuid: "f1eb71f399ba4ed0bbedf06923f39070"
format_stamp: "Formatted at 2026-09-25 22:44:26 on dist-test-slave-5vdr"
server_key: "c0f478228bd6c622d9655a38a322a6d3"
server_key_iv: "7f264957af6e39cdfce88cff91fc31ca"
server_key_version: "encryptionkey@0"
I20260925 22:44:26.948575 12500 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790376259650642-11032-0/minicluster-data/ts-2/wal/instance:
uuid: "f1eb71f399ba4ed0bbedf06923f39070"
format_stamp: "Formatted at 2026-09-25 22:44:26 on dist-test-slave-5vdr"
server_key: "c0f478228bd6c622d9655a38a322a6d3"
server_key_iv: "7f264957af6e39cdfce88cff91fc31ca"
server_key_version: "encryptionkey@0"
I20260925 22:44:26.953100 12500 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260925 22:44:26.956178 12515 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:44:26.957834 12500 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 22:44:26.958014 12500 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790376259650642-11032-0/minicluster-data/ts-2/data,/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790376259650642-11032-0/minicluster-data/ts-2/wal
uuid: "f1eb71f399ba4ed0bbedf06923f39070"
format_stamp: "Formatted at 2026-09-25 22:44:26 on dist-test-slave-5vdr"
server_key: "c0f478228bd6c622d9655a38a322a6d3"
server_key_iv: "7f264957af6e39cdfce88cff91fc31ca"
server_key_version: "encryptionkey@0"
I20260925 22:44:26.958160 12500 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790376259650642-11032-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790376259650642-11032-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790376259650642-11032-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:44:26.997622 12500 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:44:27.001557 12500 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:44:27.001849 12500 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:44:27.003499 12500 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:44:27.003609 12500 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:44:27.003692 12500 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:44:27.003739 12500 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 25 22:44:27 dist-test-slave-5vdr krb5kdc[12103](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376266, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.198.3@KRBTEST.COM for kudu/127.10.198.62@KRBTEST.COM
I20260925 22:44:27.015807 12500 rpc_server.cc:307] RPC server started. Bound to: 127.10.198.3:41659
I20260925 22:44:27.015883 12630 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.198.3:41659 every 8 connection(s)
I20260925 22:44:27.017146 12500 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790376259650642-11032-0/minicluster-data/ts-2/data/info.pb
I20260925 22:44:27.019681 12197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:44:27.004649 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.198.3:51449 (local address 127.10.198.62:42797)
0925 22:44:27.004820 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0925 22:44:27.004825 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:44:27.005861 (+ 1036us) server_negotiation.cc:407] Connection header received
0925 22:44:27.007291 (+ 1430us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:44:27.007296 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:44:27.007373 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:44:27.007510 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:44:27.009173 (+ 1663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:44:27.009731 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:44:27.011228 (+ 1497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:44:27.011549 (+ 321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:44:27.011689 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:44:27.015167 (+ 3478us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:44:27.015171 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:44:27.015173 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:44:27.015204 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:44:27.017271 (+ 2067us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:44:27.017979 (+ 708us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:44:27.017983 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:44:27.017985 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:44:27.018091 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:44:27.018497 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:44:27.018501 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:44:27.018503 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:44:27.018719 (+ 216us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:44:27.018840 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0925 22:44:27.019366 (+ 526us) server_negotiation.cc:299] Negotiation successful
0925 22:44:27.019510 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260925 22:44:27.020650 12524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:44:27.005131 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.198.62:42797 (local address 127.10.198.3:51449)
0925 22:44:27.005650 (+ 519us) negotiation.cc:107] Waiting for socket to connect
0925 22:44:27.005708 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0925 22:44:27.006943 (+ 1235us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:44:27.007748 (+ 805us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:44:27.007757 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:44:27.008214 (+ 457us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:44:27.008981 (+ 767us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:44:27.008999 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:44:27.009931 (+ 932us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:44:27.009936 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:44:27.010997 (+ 1061us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:44:27.011010 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:44:27.011754 (+ 744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:44:27.011759 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:44:27.012039 (+ 280us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:44:27.013006 (+ 967us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:44:27.013024 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:44:27.014927 (+ 1903us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:44:27.017409 (+ 2482us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:44:27.017415 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:44:27.017429 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:44:27.017825 (+ 396us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:44:27.018223 (+ 398us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:44:27.018228 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:44:27.018230 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:44:27.018382 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:44:27.018838 (+ 456us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:44:27.018845 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:44:27.019112 (+ 267us) client_negotiation.cc:769] Sending connection context
0925 22:44:27.019321 (+ 209us) client_negotiation.cc:240] Negotiation successful
0925 22:44:27.019687 (+ 366us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":298,"thread_start_us":155,"threads_started":1}
I20260925 22:44:27.022140 12631 heartbeater.cc:347] Connected to a master server at 127.10.198.62:42797
I20260925 22:44:27.022336 12631 heartbeater.cc:464] Registering TS with master...
I20260925 22:44:27.022979 12631 heartbeater.cc:511] Master 127.10.198.62:42797 requested a full tablet report, sending...
I20260925 22:44:27.024386 12147 ts_manager.cc:194] Registered new tserver with Master: f1eb71f399ba4ed0bbedf06923f39070 (127.10.198.3:41659)
I20260925 22:44:27.025192 12147 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.198.3@KRBTEST.COM'} at 127.10.198.3:51449
I20260925 22:44:27.026721 11032 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrWXm8X/build/debug/bin/kudu as pid 12500
I20260925 22:44:27.026849 11032 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790376259650642-11032-0/minicluster-data/ts-2/wal/instance
I20260925 22:44:27.027333 11032 external_mini_cluster.cc:1428] Setting key eade5208a1fcec08f34f701289088cf9
I20260925 22:44:27.030074 11032 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 22:44:27.033646 12637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:44:27.024811 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.198.62:42797 (local address 127.10.198.3:55403)
0925 22:44:27.025241 (+ 430us) negotiation.cc:107] Waiting for socket to connect
0925 22:44:27.025254 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 22:44:27.025432 (+ 178us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:44:27.025744 (+ 312us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:44:27.025746 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:44:27.025947 (+ 201us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:44:27.026071 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:44:27.026077 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:44:27.027152 (+ 1075us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:44:27.027156 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:44:27.027958 (+ 802us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:44:27.027971 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:44:27.028574 (+ 603us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:44:27.028577 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:44:27.028687 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:44:27.029377 (+ 690us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:44:27.029397 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:44:27.029863 (+ 466us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:44:27.031952 (+ 2089us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:44:27.031955 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:44:27.031957 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:44:27.032190 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:44:27.032523 (+ 333us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:44:27.032527 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:44:27.032529 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:44:27.032623 (+ 94us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:44:27.033131 (+ 508us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:44:27.033134 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:44:27.033246 (+ 112us) client_negotiation.cc:769] Sending connection context
0925 22:44:27.033342 (+ 96us) client_negotiation.cc:240] Negotiation successful
0925 22:44:27.033477 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":348,"mutex_wait_us":3,"thread_start_us":158,"threads_started":1}
I20260925 22:44:27.033877 12197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:44:27.024942 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.198.3:55403 (local address 127.10.198.62:42797)
0925 22:44:27.025098 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0925 22:44:27.025104 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 22:44:27.025359 (+ 255us) server_negotiation.cc:407] Connection header received
0925 22:44:27.025555 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:44:27.025560 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:44:27.025609 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:44:27.025714 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:44:27.026231 (+ 517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:44:27.026991 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:44:27.028083 (+ 1092us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:44:27.028451 (+ 368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:44:27.028560 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:44:27.030056 (+ 1496us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:44:27.030059 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:44:27.030061 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:44:27.030088 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:44:27.031744 (+ 1656us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:44:27.032318 (+ 574us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:44:27.032322 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:44:27.032324 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:44:27.032420 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:44:27.032720 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:44:27.032724 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:44:27.032726 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:44:27.032974 (+ 248us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:44:27.033078 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0925 22:44:27.033459 (+ 381us) server_negotiation.cc:299] Negotiation successful
0925 22:44:27.033618 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
Sep 25 22:44:27 dist-test-slave-5vdr krb5kdc[12103](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376267, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 25 22:44:27 dist-test-slave-5vdr krb5kdc[12103](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376267, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.10.198.62@KRBTEST.COM
I20260925 22:44:27.059176 12197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:44:27.049691 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49804 (local address 127.10.198.62:42797)
0925 22:44:27.049837 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0925 22:44:27.049841 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:44:27.049853 (+ 12us) server_negotiation.cc:407] Connection header received
0925 22:44:27.049895 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:44:27.049897 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:44:27.049935 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:44:27.050002 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:44:27.050852 (+ 850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:44:27.051401 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:44:27.052458 (+ 1057us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:44:27.052727 (+ 269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:44:27.052828 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:44:27.055591 (+ 2763us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:44:27.055595 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:44:27.055597 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:44:27.055634 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:44:27.057355 (+ 1721us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:44:27.057988 (+ 633us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:44:27.057991 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:44:27.057992 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:44:27.058083 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:44:27.058391 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:44:27.058393 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:44:27.058394 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:44:27.058585 (+ 191us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:44:27.058692 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0925 22:44:27.058932 (+ 240us) server_negotiation.cc:299] Negotiation successful
0925 22:44:27.059043 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260925 22:44:27.062853 12147 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:49804:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 22:44:27.066224 12147 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 22:44:27.083737 12650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:44:27.076142 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36946 (local address 127.10.198.1:38019)
0925 22:44:27.077883 (+ 1741us) server_negotiation.cc:206] Beginning negotiation
0925 22:44:27.077891 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 22:44:27.077911 (+ 20us) server_negotiation.cc:407] Connection header received
0925 22:44:27.077987 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:44:27.077992 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:44:27.078193 (+ 201us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:44:27.078628 (+ 435us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:44:27.078796 (+ 168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:44:27.079506 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:44:27.081963 (+ 2457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:44:27.082729 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:44:27.082852 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:44:27.083013 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0925 22:44:27.083420 (+ 407us) server_negotiation.cc:299] Negotiation successful
0925 22:44:27.083545 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1628,"thread_start_us":118,"threads_started":1}
I20260925 22:44:27.083755 12649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:44:27.076039 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.198.1:38019 (local address 127.0.0.1:36946)
0925 22:44:27.076716 (+ 677us) negotiation.cc:107] Waiting for socket to connect
0925 22:44:27.076782 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0925 22:44:27.076935 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:44:27.078357 (+ 1422us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:44:27.078360 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:44:27.078379 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:44:27.078479 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:44:27.078486 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:44:27.079647 (+ 1161us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:44:27.079653 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:44:27.081790 (+ 2137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:44:27.081803 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:44:27.082858 (+ 1055us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:44:27.082862 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:44:27.083098 (+ 236us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:44:27.083215 (+ 117us) client_negotiation.cc:769] Sending connection context
0925 22:44:27.083351 (+ 136us) client_negotiation.cc:240] Negotiation successful
0925 22:44:27.083585 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":466,"mutex_wait_us":468,"thread_start_us":104,"threads_started":1}
I20260925 22:44:27.084161 12648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:44:27.076444 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.198.2:34917 (local address 127.0.0.1:59018)
0925 22:44:27.077808 (+ 1364us) negotiation.cc:107] Waiting for socket to connect
0925 22:44:27.077822 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 22:44:27.077914 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:44:27.078492 (+ 578us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:44:27.078495 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:44:27.078510 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:44:27.078622 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:44:27.078628 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:44:27.080023 (+ 1395us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:44:27.080027 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:44:27.081491 (+ 1464us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:44:27.081505 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:44:27.082858 (+ 1353us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:44:27.082862 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:44:27.083098 (+ 236us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:44:27.083214 (+ 116us) client_negotiation.cc:769] Sending connection context
0925 22:44:27.083351 (+ 137us) client_negotiation.cc:240] Negotiation successful
0925 22:44:27.083585 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1291,"mutex_wait_us":345,"thread_start_us":101,"threads_started":1}
I20260925 22:44:27.084862 12653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:44:27.076958 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59018 (local address 127.10.198.2:34917)
0925 22:44:27.077329 (+ 371us) server_negotiation.cc:206] Beginning negotiation
0925 22:44:27.077344 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0925 22:44:27.077907 (+ 563us) server_negotiation.cc:407] Connection header received
0925 22:44:27.078012 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:44:27.078017 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:44:27.078193 (+ 176us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:44:27.078348 (+ 155us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:44:27.078748 (+ 400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:44:27.079902 (+ 1154us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:44:27.081665 (+ 1763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:44:27.082654 (+ 989us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:44:27.082809 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:44:27.083032 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0925 22:44:27.084569 (+ 1537us) server_negotiation.cc:299] Negotiation successful
0925 22:44:27.084704 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":258,"thread_start_us":167,"threads_started":1}
I20260925 22:44:27.084872 12651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:44:27.076141 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.198.3:41659 (local address 127.0.0.1:47012)
0925 22:44:27.077495 (+ 1354us) negotiation.cc:107] Waiting for socket to connect
0925 22:44:27.077511 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 22:44:27.077609 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:44:27.078222 (+ 613us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:44:27.078228 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:44:27.078253 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:44:27.078427 (+ 174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:44:27.078434 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:44:27.080007 (+ 1573us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:44:27.080011 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:44:27.081790 (+ 1779us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:44:27.081803 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:44:27.084426 (+ 2623us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:44:27.084430 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:44:27.084554 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:44:27.084586 (+ 32us) client_negotiation.cc:769] Sending connection context
0925 22:44:27.084649 (+ 63us) client_negotiation.cc:240] Negotiation successful
0925 22:44:27.084726 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1290,"mutex_wait_us":153,"thread_start_us":98,"threads_started":1}
I20260925 22:44:27.086462 12652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:44:27.076966 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47012 (local address 127.10.198.3:41659)
0925 22:44:27.077565 (+ 599us) server_negotiation.cc:206] Beginning negotiation
0925 22:44:27.077577 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 22:44:27.077606 (+ 29us) server_negotiation.cc:407] Connection header received
0925 22:44:27.077759 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:44:27.077765 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:44:27.078037 (+ 272us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:44:27.078222 (+ 185us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:44:27.078546 (+ 324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:44:27.079802 (+ 1256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:44:27.083108 (+ 3306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:44:27.084112 (+ 1004us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:44:27.085909 (+ 1797us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:44:27.086121 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0925 22:44:27.086182 (+ 61us) server_negotiation.cc:299] Negotiation successful
0925 22:44:27.086290 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":464,"thread_start_us":128,"threads_started":1}
I20260925 22:44:27.087894 12425 tablet_service.cc:1467] Processing CreateTablet for tablet 53c1c56424204f6dac4c6a8a7ec94e72 (DEFAULT_TABLE table=test-table [id=cc30c27fabb3419aa3ed69a5ba6711da]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 22:44:27.087980 12285 tablet_service.cc:1467] Processing CreateTablet for tablet 53c1c56424204f6dac4c6a8a7ec94e72 (DEFAULT_TABLE table=test-table [id=cc30c27fabb3419aa3ed69a5ba6711da]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 22:44:27.089414 12425 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 53c1c56424204f6dac4c6a8a7ec94e72. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:44:27.089608 12285 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 53c1c56424204f6dac4c6a8a7ec94e72. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:44:27.089830 12565 tablet_service.cc:1467] Processing CreateTablet for tablet 53c1c56424204f6dac4c6a8a7ec94e72 (DEFAULT_TABLE table=test-table [id=cc30c27fabb3419aa3ed69a5ba6711da]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 22:44:27.091055 12565 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 53c1c56424204f6dac4c6a8a7ec94e72. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:44:27.097565 12654 tablet_bootstrap.cc:492] T 53c1c56424204f6dac4c6a8a7ec94e72 P 3065b552ce554279b34368a4fec28dd2: Bootstrap starting.
I20260925 22:44:27.100304 12655 tablet_bootstrap.cc:492] T 53c1c56424204f6dac4c6a8a7ec94e72 P e58bbdc59c354af99870745d454b2612: Bootstrap starting.
I20260925 22:44:27.100898 12656 tablet_bootstrap.cc:492] T 53c1c56424204f6dac4c6a8a7ec94e72 P f1eb71f399ba4ed0bbedf06923f39070: Bootstrap starting.
I20260925 22:44:27.101330 12654 tablet_bootstrap.cc:654] T 53c1c56424204f6dac4c6a8a7ec94e72 P 3065b552ce554279b34368a4fec28dd2: Neither blocks nor log segments found. Creating new log.
I20260925 22:44:27.102376 12654 log.cc:824] T 53c1c56424204f6dac4c6a8a7ec94e72 P 3065b552ce554279b34368a4fec28dd2: Log is configured to *not* fsync() on all Append() calls
I20260925 22:44:27.102958 12655 tablet_bootstrap.cc:654] T 53c1c56424204f6dac4c6a8a7ec94e72 P e58bbdc59c354af99870745d454b2612: Neither blocks nor log segments found. Creating new log.
I20260925 22:44:27.102998 12656 tablet_bootstrap.cc:654] T 53c1c56424204f6dac4c6a8a7ec94e72 P f1eb71f399ba4ed0bbedf06923f39070: Neither blocks nor log segments found. Creating new log.
I20260925 22:44:27.103818 12656 log.cc:824] T 53c1c56424204f6dac4c6a8a7ec94e72 P f1eb71f399ba4ed0bbedf06923f39070: Log is configured to *not* fsync() on all Append() calls
I20260925 22:44:27.103895 12655 log.cc:824] T 53c1c56424204f6dac4c6a8a7ec94e72 P e58bbdc59c354af99870745d454b2612: Log is configured to *not* fsync() on all Append() calls
I20260925 22:44:27.104893 12654 tablet_bootstrap.cc:492] T 53c1c56424204f6dac4c6a8a7ec94e72 P 3065b552ce554279b34368a4fec28dd2: No bootstrap required, opened a new log
I20260925 22:44:27.105171 12654 ts_tablet_manager.cc:1397] T 53c1c56424204f6dac4c6a8a7ec94e72 P 3065b552ce554279b34368a4fec28dd2: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.000s
I20260925 22:44:27.105931 12656 tablet_bootstrap.cc:492] T 53c1c56424204f6dac4c6a8a7ec94e72 P f1eb71f399ba4ed0bbedf06923f39070: No bootstrap required, opened a new log
I20260925 22:44:27.106158 12656 ts_tablet_manager.cc:1397] T 53c1c56424204f6dac4c6a8a7ec94e72 P f1eb71f399ba4ed0bbedf06923f39070: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260925 22:44:27.106400 12655 tablet_bootstrap.cc:492] T 53c1c56424204f6dac4c6a8a7ec94e72 P e58bbdc59c354af99870745d454b2612: No bootstrap required, opened a new log
I20260925 22:44:27.106639 12655 ts_tablet_manager.cc:1397] T 53c1c56424204f6dac4c6a8a7ec94e72 P e58bbdc59c354af99870745d454b2612: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260925 22:44:27.109725 12656 raft_consensus.cc:348] T 53c1c56424204f6dac4c6a8a7ec94e72 P f1eb71f399ba4ed0bbedf06923f39070 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3065b552ce554279b34368a4fec28dd2" member_type: VOTER last_known_addr { host: "127.10.198.1" port: 38019 } } peers { permanent_uuid: "f1eb71f399ba4ed0bbedf06923f39070" member_type: VOTER last_known_addr { host: "127.10.198.3" port: 41659 } } peers { permanent_uuid: "e58bbdc59c354af99870745d454b2612" member_type: VOTER last_known_addr { host: "127.10.198.2" port: 34917 } }
I20260925 22:44:27.110052 12656 raft_consensus.cc:374] T 53c1c56424204f6dac4c6a8a7ec94e72 P f1eb71f399ba4ed0bbedf06923f39070 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:44:27.110121 12656 raft_consensus.cc:729] T 53c1c56424204f6dac4c6a8a7ec94e72 P f1eb71f399ba4ed0bbedf06923f39070 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1eb71f399ba4ed0bbedf06923f39070, State: Initialized, Role: FOLLOWER
I20260925 22:44:27.110193 12654 raft_consensus.cc:348] T 53c1c56424204f6dac4c6a8a7ec94e72 P 3065b552ce554279b34368a4fec28dd2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3065b552ce554279b34368a4fec28dd2" member_type: VOTER last_known_addr { host: "127.10.198.1" port: 38019 } } peers { permanent_uuid: "f1eb71f399ba4ed0bbedf06923f39070" member_type: VOTER last_known_addr { host: "127.10.198.3" port: 41659 } } peers { permanent_uuid: "e58bbdc59c354af99870745d454b2612" member_type: VOTER last_known_addr { host: "127.10.198.2" port: 34917 } }
I20260925 22:44:27.110497 12654 raft_consensus.cc:374] T 53c1c56424204f6dac4c6a8a7ec94e72 P 3065b552ce554279b34368a4fec28dd2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:44:27.110574 12654 raft_consensus.cc:729] T 53c1c56424204f6dac4c6a8a7ec94e72 P 3065b552ce554279b34368a4fec28dd2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3065b552ce554279b34368a4fec28dd2, State: Initialized, Role: FOLLOWER
I20260925 22:44:27.110656 12656 consensus_queue.cc:260] T 53c1c56424204f6dac4c6a8a7ec94e72 P f1eb71f399ba4ed0bbedf06923f39070 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3065b552ce554279b34368a4fec28dd2" member_type: VOTER last_known_addr { host: "127.10.198.1" port: 38019 } } peers { permanent_uuid: "f1eb71f399ba4ed0bbedf06923f39070" member_type: VOTER last_known_addr { host: "127.10.198.3" port: 41659 } } peers { permanent_uuid: "e58bbdc59c354af99870745d454b2612" member_type: VOTER last_known_addr { host: "127.10.198.2" port: 34917 } }
I20260925 22:44:27.110764 12655 raft_consensus.cc:348] T 53c1c56424204f6dac4c6a8a7ec94e72 P e58bbdc59c354af99870745d454b2612 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3065b552ce554279b34368a4fec28dd2" member_type: VOTER last_known_addr { host: "127.10.198.1" port: 38019 } } peers { permanent_uuid: "f1eb71f399ba4ed0bbedf06923f39070" member_type: VOTER last_known_addr { host: "127.10.198.3" port: 41659 } } peers { permanent_uuid: "e58bbdc59c354af99870745d454b2612" member_type: VOTER last_known_addr { host: "127.10.198.2" port: 34917 } }
I20260925 22:44:27.111019 12655 raft_consensus.cc:374] T 53c1c56424204f6dac4c6a8a7ec94e72 P e58bbdc59c354af99870745d454b2612 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:44:27.111074 12655 raft_consensus.cc:729] T 53c1c56424204f6dac4c6a8a7ec94e72 P e58bbdc59c354af99870745d454b2612 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e58bbdc59c354af99870745d454b2612, State: Initialized, Role: FOLLOWER
I20260925 22:44:27.111083 12654 consensus_queue.cc:260] T 53c1c56424204f6dac4c6a8a7ec94e72 P 3065b552ce554279b34368a4fec28dd2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3065b552ce554279b34368a4fec28dd2" member_type: VOTER last_known_addr { host: "127.10.198.1" port: 38019 } } peers { permanent_uuid: "f1eb71f399ba4ed0bbedf06923f39070" member_type: VOTER last_known_addr { host: "127.10.198.3" port: 41659 } } peers { permanent_uuid: "e58bbdc59c354af99870745d454b2612" member_type: VOTER last_known_addr { host: "127.10.198.2" port: 34917 } }
I20260925 22:44:27.111832 12351 heartbeater.cc:503] Master 127.10.198.62:42797 was elected leader, sending a full tablet report...
I20260925 22:44:27.111963 12656 ts_tablet_manager.cc:1428] T 53c1c56424204f6dac4c6a8a7ec94e72 P f1eb71f399ba4ed0bbedf06923f39070: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260925 22:44:27.112180 12654 ts_tablet_manager.cc:1428] T 53c1c56424204f6dac4c6a8a7ec94e72 P 3065b552ce554279b34368a4fec28dd2: Time spent starting tablet: real 0.007s user 0.008s sys 0.000s
I20260925 22:44:27.112335 12631 heartbeater.cc:503] Master 127.10.198.62:42797 was elected leader, sending a full tablet report...
I20260925 22:44:27.112769 12655 consensus_queue.cc:260] T 53c1c56424204f6dac4c6a8a7ec94e72 P e58bbdc59c354af99870745d454b2612 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3065b552ce554279b34368a4fec28dd2" member_type: VOTER last_known_addr { host: "127.10.198.1" port: 38019 } } peers { permanent_uuid: "f1eb71f399ba4ed0bbedf06923f39070" member_type: VOTER last_known_addr { host: "127.10.198.3" port: 41659 } } peers { permanent_uuid: "e58bbdc59c354af99870745d454b2612" member_type: VOTER last_known_addr { host: "127.10.198.2" port: 34917 } }
I20260925 22:44:27.113812 12491 heartbeater.cc:503] Master 127.10.198.62:42797 was elected leader, sending a full tablet report...
I20260925 22:44:27.113906 12655 ts_tablet_manager.cc:1428] T 53c1c56424204f6dac4c6a8a7ec94e72 P e58bbdc59c354af99870745d454b2612: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
W20260925 22:44:27.251348 12492 tablet.cc:2369] T 53c1c56424204f6dac4c6a8a7ec94e72 P e58bbdc59c354af99870745d454b2612: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 22:44:27.260118 12352 tablet.cc:2369] T 53c1c56424204f6dac4c6a8a7ec94e72 P 3065b552ce554279b34368a4fec28dd2: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 22:44:27.268410 12632 tablet.cc:2369] T 53c1c56424204f6dac4c6a8a7ec94e72 P f1eb71f399ba4ed0bbedf06923f39070: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 22:44:27.386931 12660 raft_consensus.cc:482] T 53c1c56424204f6dac4c6a8a7ec94e72 P f1eb71f399ba4ed0bbedf06923f39070 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 22:44:27.387176 12660 raft_consensus.cc:504] T 53c1c56424204f6dac4c6a8a7ec94e72 P f1eb71f399ba4ed0bbedf06923f39070 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3065b552ce554279b34368a4fec28dd2" member_type: VOTER last_known_addr { host: "127.10.198.1" port: 38019 } } peers { permanent_uuid: "f1eb71f399ba4ed0bbedf06923f39070" member_type: VOTER last_known_addr { host: "127.10.198.3" port: 41659 } } peers { permanent_uuid: "e58bbdc59c354af99870745d454b2612" member_type: VOTER last_known_addr { host: "127.10.198.2" port: 34917 } }
I20260925 22:44:27.388513 12660 leader_election.cc:288] T 53c1c56424204f6dac4c6a8a7ec94e72 P f1eb71f399ba4ed0bbedf06923f39070 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3065b552ce554279b34368a4fec28dd2 (127.10.198.1:38019), e58bbdc59c354af99870745d454b2612 (127.10.198.2:34917)
I20260925 22:44:27.392915 12524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:44:27.388929 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.198.2:34917 (local address 127.10.198.3:46875)
0925 22:44:27.389084 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0925 22:44:27.389104 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0925 22:44:27.389226 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:44:27.389522 (+ 296us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:44:27.389529 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:44:27.389553 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:44:27.389729 (+ 176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:44:27.389737 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:44:27.390611 (+ 874us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:44:27.390613 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:44:27.391666 (+ 1053us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:44:27.391678 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:44:27.392539 (+ 861us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:44:27.392541 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:44:27.392630 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:44:27.392653 (+ 23us) client_negotiation.cc:769] Sending connection context
0925 22:44:27.392725 (+ 72us) client_negotiation.cc:240] Negotiation successful
0925 22:44:27.392793 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":56}
I20260925 22:44:27.393002 12653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:44:27.389067 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.198.3:46875 (local address 127.10.198.2:34917)
0925 22:44:27.389224 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0925 22:44:27.389232 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 22:44:27.389256 (+ 24us) server_negotiation.cc:407] Connection header received
0925 22:44:27.389329 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:44:27.389332 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:44:27.389394 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:44:27.389489 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:44:27.389868 (+ 379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:44:27.390424 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:44:27.391895 (+ 1471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:44:27.392392 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:44:27.392565 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:44:27.392636 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0925 22:44:27.392763 (+ 127us) server_negotiation.cc:299] Negotiation successful
0925 22:44:27.392854 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260925 22:44:27.393792 12650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:44:27.389066 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.198.3:51639 (local address 127.10.198.1:38019)
0925 22:44:27.389219 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0925 22:44:27.389224 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:44:27.389650 (+ 426us) server_negotiation.cc:407] Connection header received
0925 22:44:27.389725 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:44:27.389727 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:44:27.389771 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:44:27.389875 (+ 104us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:44:27.390547 (+ 672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:44:27.391091 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:44:27.392385 (+ 1294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:44:27.393085 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:44:27.393239 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:44:27.393306 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0925 22:44:27.393589 (+ 283us) server_negotiation.cc:299] Negotiation successful
0925 22:44:27.393673 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260925 22:44:27.393819 12663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:44:27.388929 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.198.1:38019 (local address 127.10.198.3:51639)
0925 22:44:27.389519 (+ 590us) negotiation.cc:107] Waiting for socket to connect
0925 22:44:27.389533 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 22:44:27.389634 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:44:27.389910 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:44:27.389912 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:44:27.389927 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:44:27.390387 (+ 460us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:44:27.390394 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:44:27.391243 (+ 849us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:44:27.391247 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:44:27.392238 (+ 991us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:44:27.392247 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:44:27.393308 (+ 1061us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:44:27.393313 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:44:27.393423 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:44:27.393445 (+ 22us) client_negotiation.cc:769] Sending connection context
0925 22:44:27.393557 (+ 112us) client_negotiation.cc:240] Negotiation successful
0925 22:44:27.393641 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":488,"mutex_wait_us":161,"thread_start_us":142,"threads_started":1}
I20260925 22:44:27.393791 12445 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53c1c56424204f6dac4c6a8a7ec94e72" candidate_uuid: "f1eb71f399ba4ed0bbedf06923f39070" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e58bbdc59c354af99870745d454b2612" is_pre_election: true
I20260925 22:44:27.394181 12445 raft_consensus.cc:2454] T 53c1c56424204f6dac4c6a8a7ec94e72 P e58bbdc59c354af99870745d454b2612 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f1eb71f399ba4ed0bbedf06923f39070 in term 0.
I20260925 22:44:27.394562 12305 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53c1c56424204f6dac4c6a8a7ec94e72" candidate_uuid: "f1eb71f399ba4ed0bbedf06923f39070" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3065b552ce554279b34368a4fec28dd2" is_pre_election: true
I20260925 22:44:27.394876 12305 raft_consensus.cc:2454] T 53c1c56424204f6dac4c6a8a7ec94e72 P 3065b552ce554279b34368a4fec28dd2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f1eb71f399ba4ed0bbedf06923f39070 in term 0.
I20260925 22:44:27.394907 12518 leader_election.cc:302] T 53c1c56424204f6dac4c6a8a7ec94e72 P f1eb71f399ba4ed0bbedf06923f39070 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e58bbdc59c354af99870745d454b2612, f1eb71f399ba4ed0bbedf06923f39070; no voters:
I20260925 22:44:27.395277 12660 raft_consensus.cc:2781] T 53c1c56424204f6dac4c6a8a7ec94e72 P f1eb71f399ba4ed0bbedf06923f39070 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 22:44:27.395372 12660 raft_consensus.cc:482] T 53c1c56424204f6dac4c6a8a7ec94e72 P f1eb71f399ba4ed0bbedf06923f39070 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 22:44:27.395429 12660 raft_consensus.cc:3037] T 53c1c56424204f6dac4c6a8a7ec94e72 P f1eb71f399ba4ed0bbedf06923f39070 [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:44:27.396627 12660 raft_consensus.cc:504] T 53c1c56424204f6dac4c6a8a7ec94e72 P f1eb71f399ba4ed0bbedf06923f39070 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3065b552ce554279b34368a4fec28dd2" member_type: VOTER last_known_addr { host: "127.10.198.1" port: 38019 } } peers { permanent_uuid: "f1eb71f399ba4ed0bbedf06923f39070" member_type: VOTER last_known_addr { host: "127.10.198.3" port: 41659 } } peers { permanent_uuid: "e58bbdc59c354af99870745d454b2612" member_type: VOTER last_known_addr { host: "127.10.198.2" port: 34917 } }
I20260925 22:44:27.397095 12660 leader_election.cc:288] T 53c1c56424204f6dac4c6a8a7ec94e72 P f1eb71f399ba4ed0bbedf06923f39070 [CANDIDATE]: Term 1 election: Requested vote from peers 3065b552ce554279b34368a4fec28dd2 (127.10.198.1:38019), e58bbdc59c354af99870745d454b2612 (127.10.198.2:34917)
I20260925 22:44:27.397588 12305 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53c1c56424204f6dac4c6a8a7ec94e72" candidate_uuid: "f1eb71f399ba4ed0bbedf06923f39070" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3065b552ce554279b34368a4fec28dd2"
I20260925 22:44:27.397769 12305 raft_consensus.cc:3037] T 53c1c56424204f6dac4c6a8a7ec94e72 P 3065b552ce554279b34368a4fec28dd2 [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:44:27.398954 12305 raft_consensus.cc:2454] T 53c1c56424204f6dac4c6a8a7ec94e72 P 3065b552ce554279b34368a4fec28dd2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f1eb71f399ba4ed0bbedf06923f39070 in term 1.
I20260925 22:44:27.399345 12445 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53c1c56424204f6dac4c6a8a7ec94e72" candidate_uuid: "f1eb71f399ba4ed0bbedf06923f39070" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e58bbdc59c354af99870745d454b2612"
I20260925 22:44:27.399478 12517 leader_election.cc:302] T 53c1c56424204f6dac4c6a8a7ec94e72 P f1eb71f399ba4ed0bbedf06923f39070 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3065b552ce554279b34368a4fec28dd2, f1eb71f399ba4ed0bbedf06923f39070; no voters:
I20260925 22:44:27.399521 12445 raft_consensus.cc:3037] T 53c1c56424204f6dac4c6a8a7ec94e72 P e58bbdc59c354af99870745d454b2612 [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:44:27.399739 12660 raft_consensus.cc:2781] T 53c1c56424204f6dac4c6a8a7ec94e72 P f1eb71f399ba4ed0bbedf06923f39070 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 22:44:27.400018 12660 raft_consensus.cc:686] T 53c1c56424204f6dac4c6a8a7ec94e72 P f1eb71f399ba4ed0bbedf06923f39070 [term 1 LEADER]: Becoming Leader. State: Replica: f1eb71f399ba4ed0bbedf06923f39070, State: Running, Role: LEADER
I20260925 22:44:27.400446 12660 consensus_queue.cc:237] T 53c1c56424204f6dac4c6a8a7ec94e72 P f1eb71f399ba4ed0bbedf06923f39070 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3065b552ce554279b34368a4fec28dd2" member_type: VOTER last_known_addr { host: "127.10.198.1" port: 38019 } } peers { permanent_uuid: "f1eb71f399ba4ed0bbedf06923f39070" member_type: VOTER last_known_addr { host: "127.10.198.3" port: 41659 } } peers { permanent_uuid: "e58bbdc59c354af99870745d454b2612" member_type: VOTER last_known_addr { host: "127.10.198.2" port: 34917 } }
I20260925 22:44:27.400851 12445 raft_consensus.cc:2454] T 53c1c56424204f6dac4c6a8a7ec94e72 P e58bbdc59c354af99870745d454b2612 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f1eb71f399ba4ed0bbedf06923f39070 in term 1.
I20260925 22:44:27.404238 12146 catalog_manager.cc:5495] T 53c1c56424204f6dac4c6a8a7ec94e72 P f1eb71f399ba4ed0bbedf06923f39070 reported cstate change: term changed from 0 to 1, leader changed from <none> to f1eb71f399ba4ed0bbedf06923f39070 (127.10.198.3). New cstate: current_term: 1 leader_uuid: "f1eb71f399ba4ed0bbedf06923f39070" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3065b552ce554279b34368a4fec28dd2" member_type: VOTER last_known_addr { host: "127.10.198.1" port: 38019 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f1eb71f399ba4ed0bbedf06923f39070" member_type: VOTER last_known_addr { host: "127.10.198.3" port: 41659 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e58bbdc59c354af99870745d454b2612" member_type: VOTER last_known_addr { host: "127.10.198.2" port: 34917 } health_report { overall_health: UNKNOWN } } }
I20260925 22:44:27.457043 12652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:44:27.453095 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47014 (local address 127.10.198.3:41659)
0925 22:44:27.453243 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0925 22:44:27.453247 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:44:27.453299 (+ 52us) server_negotiation.cc:407] Connection header received
0925 22:44:27.453520 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:44:27.453522 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:44:27.453580 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:44:27.453676 (+ 96us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 22:44:27.453927 (+ 251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:44:27.454460 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:44:27.455403 (+ 943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:44:27.455622 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:44:27.455713 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:44:27.456104 (+ 391us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 22:44:27.456588 (+ 484us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 22:44:27.456687 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0925 22:44:27.456866 (+ 179us) server_negotiation.cc:299] Negotiation successful
0925 22:44:27.456929 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260925 22:44:27.463595 12445 raft_consensus.cc:1261] T 53c1c56424204f6dac4c6a8a7ec94e72 P e58bbdc59c354af99870745d454b2612 [term 1 FOLLOWER]: Refusing update from remote peer f1eb71f399ba4ed0bbedf06923f39070: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 22:44:27.463598 12305 raft_consensus.cc:1261] T 53c1c56424204f6dac4c6a8a7ec94e72 P 3065b552ce554279b34368a4fec28dd2 [term 1 FOLLOWER]: Refusing update from remote peer f1eb71f399ba4ed0bbedf06923f39070: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 22:44:27.464484 12660 consensus_queue.cc:1035] T 53c1c56424204f6dac4c6a8a7ec94e72 P f1eb71f399ba4ed0bbedf06923f39070 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3065b552ce554279b34368a4fec28dd2" member_type: VOTER last_known_addr { host: "127.10.198.1" port: 38019 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 22:44:27.464720 12664 consensus_queue.cc:1035] T 53c1c56424204f6dac4c6a8a7ec94e72 P f1eb71f399ba4ed0bbedf06923f39070 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e58bbdc59c354af99870745d454b2612" member_type: VOTER last_known_addr { host: "127.10.198.2" port: 34917 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 22:44:27.472846 12667 mvcc.cc:205] Tried to move back new op lower bound from 7333381191522480128 to 7333381191275708416. Current Snapshot: MvccSnapshot[applied={T|T < 7333381191522480128}]
I20260925 22:44:27.475095 12668 mvcc.cc:205] Tried to move back new op lower bound from 7333381191522480128 to 7333381191275708416. Current Snapshot: MvccSnapshot[applied={T|T < 7333381191522480128}]
I20260925 22:44:27.486756 12148 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:49804:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 22:44:27.486941 12148 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:49804:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 22:44:27.489872 12148 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d5451cd6c49743beb1c5b14b5163a164: Sending DeleteTablet for 3 replicas of tablet 53c1c56424204f6dac4c6a8a7ec94e72
I20260925 22:44:27.490830 12285 tablet_service.cc:1514] Processing DeleteTablet for tablet 53c1c56424204f6dac4c6a8a7ec94e72 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 22:44:27 UTC) from {username='kudu', principal='kudu/127.10.198.62@KRBTEST.COM'} at 127.0.0.1:36946
I20260925 22:44:27.490825 12425 tablet_service.cc:1514] Processing DeleteTablet for tablet 53c1c56424204f6dac4c6a8a7ec94e72 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 22:44:27 UTC) from {username='kudu', principal='kudu/127.10.198.62@KRBTEST.COM'} at 127.0.0.1:59018
I20260925 22:44:27.490844 12565 tablet_service.cc:1514] Processing DeleteTablet for tablet 53c1c56424204f6dac4c6a8a7ec94e72 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 22:44:27 UTC) from {username='kudu', principal='kudu/127.10.198.62@KRBTEST.COM'} at 127.0.0.1:47012
I20260925 22:44:27.491422 12678 tablet_replica.cc:330] T 53c1c56424204f6dac4c6a8a7ec94e72 P 3065b552ce554279b34368a4fec28dd2: stopping tablet replica
I20260925 22:44:27.491611 12677 tablet_replica.cc:330] T 53c1c56424204f6dac4c6a8a7ec94e72 P f1eb71f399ba4ed0bbedf06923f39070: stopping tablet replica
I20260925 22:44:27.491833 12678 raft_consensus.cc:2229] T 53c1c56424204f6dac4c6a8a7ec94e72 P 3065b552ce554279b34368a4fec28dd2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 22:44:27.492090 12677 raft_consensus.cc:2229] T 53c1c56424204f6dac4c6a8a7ec94e72 P f1eb71f399ba4ed0bbedf06923f39070 [term 1 LEADER]: Raft consensus shutting down.
I20260925 22:44:27.492229 12678 raft_consensus.cc:2258] T 53c1c56424204f6dac4c6a8a7ec94e72 P 3065b552ce554279b34368a4fec28dd2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 22:44:27.492602 12677 raft_consensus.cc:2258] T 53c1c56424204f6dac4c6a8a7ec94e72 P f1eb71f399ba4ed0bbedf06923f39070 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 22:44:27.492808 12676 tablet_replica.cc:330] T 53c1c56424204f6dac4c6a8a7ec94e72 P e58bbdc59c354af99870745d454b2612: stopping tablet replica
I20260925 22:44:27.492964 12146 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:49804:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260925 22:44:27.493122 12676 raft_consensus.cc:2229] T 53c1c56424204f6dac4c6a8a7ec94e72 P e58bbdc59c354af99870745d454b2612 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 22:44:27.493449 12676 raft_consensus.cc:2258] T 53c1c56424204f6dac4c6a8a7ec94e72 P e58bbdc59c354af99870745d454b2612 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260925 22:44:27.493449 12146 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 22:44:27.494796 12677 ts_tablet_manager.cc:1905] T 53c1c56424204f6dac4c6a8a7ec94e72 P f1eb71f399ba4ed0bbedf06923f39070: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 22:44:27.494797 12676 ts_tablet_manager.cc:1905] T 53c1c56424204f6dac4c6a8a7ec94e72 P e58bbdc59c354af99870745d454b2612: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 22:44:27.497951 12677 ts_tablet_manager.cc:1918] T 53c1c56424204f6dac4c6a8a7ec94e72 P f1eb71f399ba4ed0bbedf06923f39070: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 22:44:27.498008 12676 ts_tablet_manager.cc:1918] T 53c1c56424204f6dac4c6a8a7ec94e72 P e58bbdc59c354af99870745d454b2612: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 22:44:27.498121 12676 log.cc:1196] T 53c1c56424204f6dac4c6a8a7ec94e72 P e58bbdc59c354af99870745d454b2612: Deleting WAL directory at /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790376259650642-11032-0/minicluster-data/ts-1/wal/wals/53c1c56424204f6dac4c6a8a7ec94e72
I20260925 22:44:27.498260 12677 log.cc:1196] T 53c1c56424204f6dac4c6a8a7ec94e72 P f1eb71f399ba4ed0bbedf06923f39070: Deleting WAL directory at /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790376259650642-11032-0/minicluster-data/ts-2/wal/wals/53c1c56424204f6dac4c6a8a7ec94e72
I20260925 22:44:27.498476 12676 ts_tablet_manager.cc:1939] T 53c1c56424204f6dac4c6a8a7ec94e72 P e58bbdc59c354af99870745d454b2612: Deleting consensus metadata
I20260925 22:44:27.498719 12677 ts_tablet_manager.cc:1939] T 53c1c56424204f6dac4c6a8a7ec94e72 P f1eb71f399ba4ed0bbedf06923f39070: Deleting consensus metadata
I20260925 22:44:27.499960 12132 catalog_manager.cc:4841] TS f1eb71f399ba4ed0bbedf06923f39070 (127.10.198.3:41659): tablet 53c1c56424204f6dac4c6a8a7ec94e72 (table test-table [id=cc30c27fabb3419aa3ed69a5ba6711da]) successfully deleted
I20260925 22:44:27.499960 12134 catalog_manager.cc:4841] TS e58bbdc59c354af99870745d454b2612 (127.10.198.2:34917): tablet 53c1c56424204f6dac4c6a8a7ec94e72 (table test-table [id=cc30c27fabb3419aa3ed69a5ba6711da]) successfully deleted
I20260925 22:44:27.500661 12285 tablet_service.cc:1467] Processing CreateTablet for tablet 9a096b4ae2d944138a7d928c3fd1bc71 (DEFAULT_TABLE table=test-table [id=200353dfa3904226924ce522d7a61ab9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 22:44:27.500936 12425 tablet_service.cc:1467] Processing CreateTablet for tablet 9a096b4ae2d944138a7d928c3fd1bc71 (DEFAULT_TABLE table=test-table [id=200353dfa3904226924ce522d7a61ab9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 22:44:27.501008 12285 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9a096b4ae2d944138a7d928c3fd1bc71. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:44:27.501204 12425 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9a096b4ae2d944138a7d928c3fd1bc71. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:44:27.502192 12678 ts_tablet_manager.cc:1905] T 53c1c56424204f6dac4c6a8a7ec94e72 P 3065b552ce554279b34368a4fec28dd2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 22:44:27.503688 12655 tablet_bootstrap.cc:492] T 9a096b4ae2d944138a7d928c3fd1bc71 P e58bbdc59c354af99870745d454b2612: Bootstrap starting.
I20260925 22:44:27.503696 12654 tablet_bootstrap.cc:492] T 9a096b4ae2d944138a7d928c3fd1bc71 P 3065b552ce554279b34368a4fec28dd2: Bootstrap starting.
I20260925 22:44:27.504383 12565 tablet_service.cc:1467] Processing CreateTablet for tablet 9a096b4ae2d944138a7d928c3fd1bc71 (DEFAULT_TABLE table=test-table [id=200353dfa3904226924ce522d7a61ab9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 22:44:27.504673 12565 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9a096b4ae2d944138a7d928c3fd1bc71. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:44:27.504814 12654 tablet_bootstrap.cc:654] T 9a096b4ae2d944138a7d928c3fd1bc71 P 3065b552ce554279b34368a4fec28dd2: Neither blocks nor log segments found. Creating new log.
I20260925 22:44:27.506443 12654 tablet_bootstrap.cc:492] T 9a096b4ae2d944138a7d928c3fd1bc71 P 3065b552ce554279b34368a4fec28dd2: No bootstrap required, opened a new log
I20260925 22:44:27.506527 12654 ts_tablet_manager.cc:1397] T 9a096b4ae2d944138a7d928c3fd1bc71 P 3065b552ce554279b34368a4fec28dd2: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260925 22:44:27.507156 12656 tablet_bootstrap.cc:492] T 9a096b4ae2d944138a7d928c3fd1bc71 P f1eb71f399ba4ed0bbedf06923f39070: Bootstrap starting.
I20260925 22:44:27.507474 12654 raft_consensus.cc:348] T 9a096b4ae2d944138a7d928c3fd1bc71 P 3065b552ce554279b34368a4fec28dd2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3065b552ce554279b34368a4fec28dd2" member_type: VOTER last_known_addr { host: "127.10.198.1" port: 38019 } } peers { permanent_uuid: "e58bbdc59c354af99870745d454b2612" member_type: VOTER last_known_addr { host: "127.10.198.2" port: 34917 } } peers { permanent_uuid: "f1eb71f399ba4ed0bbedf06923f39070" member_type: VOTER last_known_addr { host: "127.10.198.3" port: 41659 } }
I20260925 22:44:27.507599 12654 raft_consensus.cc:374] T 9a096b4ae2d944138a7d928c3fd1bc71 P 3065b552ce554279b34368a4fec28dd2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:44:27.507633 12654 raft_consensus.cc:729] T 9a096b4ae2d944138a7d928c3fd1bc71 P 3065b552ce554279b34368a4fec28dd2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3065b552ce554279b34368a4fec28dd2, State: Initialized, Role: FOLLOWER
I20260925 22:44:27.507761 12654 consensus_queue.cc:260] T 9a096b4ae2d944138a7d928c3fd1bc71 P 3065b552ce554279b34368a4fec28dd2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3065b552ce554279b34368a4fec28dd2" member_type: VOTER last_known_addr { host: "127.10.198.1" port: 38019 } } peers { permanent_uuid: "e58bbdc59c354af99870745d454b2612" member_type: VOTER last_known_addr { host: "127.10.198.2" port: 34917 } } peers { permanent_uuid: "f1eb71f399ba4ed0bbedf06923f39070" member_type: VOTER last_known_addr { host: "127.10.198.3" port: 41659 } }
I20260925 22:44:27.508046 12654 ts_tablet_manager.cc:1428] T 9a096b4ae2d944138a7d928c3fd1bc71 P 3065b552ce554279b34368a4fec28dd2: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20260925 22:44:27.508190 12656 tablet_bootstrap.cc:654] T 9a096b4ae2d944138a7d928c3fd1bc71 P f1eb71f399ba4ed0bbedf06923f39070: Neither blocks nor log segments found. Creating new log.
I20260925 22:44:27.510143 12655 tablet_bootstrap.cc:654] T 9a096b4ae2d944138a7d928c3fd1bc71 P e58bbdc59c354af99870745d454b2612: Neither blocks nor log segments found. Creating new log.
I20260925 22:44:27.511752 12655 tablet_bootstrap.cc:492] T 9a096b4ae2d944138a7d928c3fd1bc71 P e58bbdc59c354af99870745d454b2612: No bootstrap required, opened a new log
I20260925 22:44:27.511909 12655 ts_tablet_manager.cc:1397] T 9a096b4ae2d944138a7d928c3fd1bc71 P e58bbdc59c354af99870745d454b2612: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.000s
I20260925 22:44:27.513080 12655 raft_consensus.cc:348] T 9a096b4ae2d944138a7d928c3fd1bc71 P e58bbdc59c354af99870745d454b2612 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3065b552ce554279b34368a4fec28dd2" member_type: VOTER last_known_addr { host: "127.10.198.1" port: 38019 } } peers { permanent_uuid: "e58bbdc59c354af99870745d454b2612" member_type: VOTER last_known_addr { host: "127.10.198.2" port: 34917 } } peers { permanent_uuid: "f1eb71f399ba4ed0bbedf06923f39070" member_type: VOTER last_known_addr { host: "127.10.198.3" port: 41659 } }
I20260925 22:44:27.513263 12655 raft_consensus.cc:374] T 9a096b4ae2d944138a7d928c3fd1bc71 P e58bbdc59c354af99870745d454b2612 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:44:27.513324 12655 raft_consensus.cc:729] T 9a096b4ae2d944138a7d928c3fd1bc71 P e58bbdc59c354af99870745d454b2612 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e58bbdc59c354af99870745d454b2612, State: Initialized, Role: FOLLOWER
I20260925 22:44:27.513561 12655 consensus_queue.cc:260] T 9a096b4ae2d944138a7d928c3fd1bc71 P e58bbdc59c354af99870745d454b2612 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3065b552ce554279b34368a4fec28dd2" member_type: VOTER last_known_addr { host: "127.10.198.1" port: 38019 } } peers { permanent_uuid: "e58bbdc59c354af99870745d454b2612" member_type: VOTER last_known_addr { host: "127.10.198.2" port: 34917 } } peers { permanent_uuid: "f1eb71f399ba4ed0bbedf06923f39070" member_type: VOTER last_known_addr { host: "127.10.198.3" port: 41659 } }
I20260925 22:44:27.516443 12655 ts_tablet_manager.cc:1428] T 9a096b4ae2d944138a7d928c3fd1bc71 P e58bbdc59c354af99870745d454b2612: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20260925 22:44:27.518281 12656 tablet_bootstrap.cc:492] T 9a096b4ae2d944138a7d928c3fd1bc71 P f1eb71f399ba4ed0bbedf06923f39070: No bootstrap required, opened a new log
I20260925 22:44:27.518409 12656 ts_tablet_manager.cc:1397] T 9a096b4ae2d944138a7d928c3fd1bc71 P f1eb71f399ba4ed0bbedf06923f39070: Time spent bootstrapping tablet: real 0.011s user 0.002s sys 0.000s
I20260925 22:44:27.518903 12656 raft_consensus.cc:348] T 9a096b4ae2d944138a7d928c3fd1bc71 P f1eb71f399ba4ed0bbedf06923f39070 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3065b552ce554279b34368a4fec28dd2" member_type: VOTER last_known_addr { host: "127.10.198.1" port: 38019 } } peers { permanent_uuid: "e58bbdc59c354af99870745d454b2612" member_type: VOTER last_known_addr { host: "127.10.198.2" port: 34917 } } peers { permanent_uuid: "f1eb71f399ba4ed0bbedf06923f39070" member_type: VOTER last_known_addr { host: "127.10.198.3" port: 41659 } }
I20260925 22:44:27.519037 12656 raft_consensus.cc:374] T 9a096b4ae2d944138a7d928c3fd1bc71 P f1eb71f399ba4ed0bbedf06923f39070 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:44:27.519065 12656 raft_consensus.cc:729] T 9a096b4ae2d944138a7d928c3fd1bc71 P f1eb71f399ba4ed0bbedf06923f39070 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1eb71f399ba4ed0bbedf06923f39070, State: Initialized, Role: FOLLOWER
I20260925 22:44:27.519203 12656 consensus_queue.cc:260] T 9a096b4ae2d944138a7d928c3fd1bc71 P f1eb71f399ba4ed0bbedf06923f39070 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3065b552ce554279b34368a4fec28dd2" member_type: VOTER last_known_addr { host: "127.10.198.1" port: 38019 } } peers { permanent_uuid: "e58bbdc59c354af99870745d454b2612" member_type: VOTER last_known_addr { host: "127.10.198.2" port: 34917 } } peers { permanent_uuid: "f1eb71f399ba4ed0bbedf06923f39070" member_type: VOTER last_known_addr { host: "127.10.198.3" port: 41659 } }
I20260925 22:44:27.519487 12656 ts_tablet_manager.cc:1428] T 9a096b4ae2d944138a7d928c3fd1bc71 P f1eb71f399ba4ed0bbedf06923f39070: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260925 22:44:27.521308 12678 ts_tablet_manager.cc:1918] T 53c1c56424204f6dac4c6a8a7ec94e72 P 3065b552ce554279b34368a4fec28dd2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 22:44:27.521451 12678 log.cc:1196] T 53c1c56424204f6dac4c6a8a7ec94e72 P 3065b552ce554279b34368a4fec28dd2: Deleting WAL directory at /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790376259650642-11032-0/minicluster-data/ts-0/wal/wals/53c1c56424204f6dac4c6a8a7ec94e72
I20260925 22:44:27.521874 12678 ts_tablet_manager.cc:1939] T 53c1c56424204f6dac4c6a8a7ec94e72 P 3065b552ce554279b34368a4fec28dd2: Deleting consensus metadata
I20260925 22:44:27.523411 12133 catalog_manager.cc:4841] TS 3065b552ce554279b34368a4fec28dd2 (127.10.198.1:38019): tablet 53c1c56424204f6dac4c6a8a7ec94e72 (table test-table [id=cc30c27fabb3419aa3ed69a5ba6711da]) successfully deleted
I20260925 22:44:27.540622 12664 raft_consensus.cc:482] T 9a096b4ae2d944138a7d928c3fd1bc71 P f1eb71f399ba4ed0bbedf06923f39070 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 22:44:27.540776 12664 raft_consensus.cc:504] T 9a096b4ae2d944138a7d928c3fd1bc71 P f1eb71f399ba4ed0bbedf06923f39070 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3065b552ce554279b34368a4fec28dd2" member_type: VOTER last_known_addr { host: "127.10.198.1" port: 38019 } } peers { permanent_uuid: "e58bbdc59c354af99870745d454b2612" member_type: VOTER last_known_addr { host: "127.10.198.2" port: 34917 } } peers { permanent_uuid: "f1eb71f399ba4ed0bbedf06923f39070" member_type: VOTER last_known_addr { host: "127.10.198.3" port: 41659 } }
I20260925 22:44:27.541251 12664 leader_election.cc:288] T 9a096b4ae2d944138a7d928c3fd1bc71 P f1eb71f399ba4ed0bbedf06923f39070 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3065b552ce554279b34368a4fec28dd2 (127.10.198.1:38019), e58bbdc59c354af99870745d454b2612 (127.10.198.2:34917)
I20260925 22:44:27.541780 12305 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a096b4ae2d944138a7d928c3fd1bc71" candidate_uuid: "f1eb71f399ba4ed0bbedf06923f39070" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3065b552ce554279b34368a4fec28dd2" is_pre_election: true
I20260925 22:44:27.541780 12445 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a096b4ae2d944138a7d928c3fd1bc71" candidate_uuid: "f1eb71f399ba4ed0bbedf06923f39070" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e58bbdc59c354af99870745d454b2612" is_pre_election: true
I20260925 22:44:27.541958 12445 raft_consensus.cc:2454] T 9a096b4ae2d944138a7d928c3fd1bc71 P e58bbdc59c354af99870745d454b2612 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f1eb71f399ba4ed0bbedf06923f39070 in term 0.
I20260925 22:44:27.541962 12305 raft_consensus.cc:2454] T 9a096b4ae2d944138a7d928c3fd1bc71 P 3065b552ce554279b34368a4fec28dd2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f1eb71f399ba4ed0bbedf06923f39070 in term 0.
I20260925 22:44:27.542294 12518 leader_election.cc:302] T 9a096b4ae2d944138a7d928c3fd1bc71 P f1eb71f399ba4ed0bbedf06923f39070 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e58bbdc59c354af99870745d454b2612, f1eb71f399ba4ed0bbedf06923f39070; no voters:
I20260925 22:44:27.542554 12664 raft_consensus.cc:2781] T 9a096b4ae2d944138a7d928c3fd1bc71 P f1eb71f399ba4ed0bbedf06923f39070 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 22:44:27.542606 12664 raft_consensus.cc:482] T 9a096b4ae2d944138a7d928c3fd1bc71 P f1eb71f399ba4ed0bbedf06923f39070 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 22:44:27.542634 12664 raft_consensus.cc:3037] T 9a096b4ae2d944138a7d928c3fd1bc71 P f1eb71f399ba4ed0bbedf06923f39070 [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:44:27.543720 12664 raft_consensus.cc:504] T 9a096b4ae2d944138a7d928c3fd1bc71 P f1eb71f399ba4ed0bbedf06923f39070 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3065b552ce554279b34368a4fec28dd2" member_type: VOTER last_known_addr { host: "127.10.198.1" port: 38019 } } peers { permanent_uuid: "e58bbdc59c354af99870745d454b2612" member_type: VOTER last_known_addr { host: "127.10.198.2" port: 34917 } } peers { permanent_uuid: "f1eb71f399ba4ed0bbedf06923f39070" member_type: VOTER last_known_addr { host: "127.10.198.3" port: 41659 } }
I20260925 22:44:27.544489 12664 leader_election.cc:288] T 9a096b4ae2d944138a7d928c3fd1bc71 P f1eb71f399ba4ed0bbedf06923f39070 [CANDIDATE]: Term 1 election: Requested vote from peers 3065b552ce554279b34368a4fec28dd2 (127.10.198.1:38019), e58bbdc59c354af99870745d454b2612 (127.10.198.2:34917)
I20260925 22:44:27.544902 12445 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a096b4ae2d944138a7d928c3fd1bc71" candidate_uuid: "f1eb71f399ba4ed0bbedf06923f39070" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e58bbdc59c354af99870745d454b2612"
I20260925 22:44:27.544903 12305 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a096b4ae2d944138a7d928c3fd1bc71" candidate_uuid: "f1eb71f399ba4ed0bbedf06923f39070" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3065b552ce554279b34368a4fec28dd2"
I20260925 22:44:27.545073 12305 raft_consensus.cc:3037] T 9a096b4ae2d944138a7d928c3fd1bc71 P 3065b552ce554279b34368a4fec28dd2 [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:44:27.545073 12445 raft_consensus.cc:3037] T 9a096b4ae2d944138a7d928c3fd1bc71 P e58bbdc59c354af99870745d454b2612 [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:44:27.546121 12445 raft_consensus.cc:2454] T 9a096b4ae2d944138a7d928c3fd1bc71 P e58bbdc59c354af99870745d454b2612 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f1eb71f399ba4ed0bbedf06923f39070 in term 1.
I20260925 22:44:27.546288 12305 raft_consensus.cc:2454] T 9a096b4ae2d944138a7d928c3fd1bc71 P 3065b552ce554279b34368a4fec28dd2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f1eb71f399ba4ed0bbedf06923f39070 in term 1.
I20260925 22:44:27.546671 12517 leader_election.cc:302] T 9a096b4ae2d944138a7d928c3fd1bc71 P f1eb71f399ba4ed0bbedf06923f39070 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3065b552ce554279b34368a4fec28dd2, f1eb71f399ba4ed0bbedf06923f39070; no voters:
I20260925 22:44:27.546952 12664 raft_consensus.cc:2781] T 9a096b4ae2d944138a7d928c3fd1bc71 P f1eb71f399ba4ed0bbedf06923f39070 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 22:44:27.547135 12664 raft_consensus.cc:686] T 9a096b4ae2d944138a7d928c3fd1bc71 P f1eb71f399ba4ed0bbedf06923f39070 [term 1 LEADER]: Becoming Leader. State: Replica: f1eb71f399ba4ed0bbedf06923f39070, State: Running, Role: LEADER
I20260925 22:44:27.547338 12664 consensus_queue.cc:237] T 9a096b4ae2d944138a7d928c3fd1bc71 P f1eb71f399ba4ed0bbedf06923f39070 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3065b552ce554279b34368a4fec28dd2" member_type: VOTER last_known_addr { host: "127.10.198.1" port: 38019 } } peers { permanent_uuid: "e58bbdc59c354af99870745d454b2612" member_type: VOTER last_known_addr { host: "127.10.198.2" port: 34917 } } peers { permanent_uuid: "f1eb71f399ba4ed0bbedf06923f39070" member_type: VOTER last_known_addr { host: "127.10.198.3" port: 41659 } }
I20260925 22:44:27.549268 12146 catalog_manager.cc:5495] T 9a096b4ae2d944138a7d928c3fd1bc71 P f1eb71f399ba4ed0bbedf06923f39070 reported cstate change: term changed from 0 to 1, leader changed from <none> to f1eb71f399ba4ed0bbedf06923f39070 (127.10.198.3). New cstate: current_term: 1 leader_uuid: "f1eb71f399ba4ed0bbedf06923f39070" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3065b552ce554279b34368a4fec28dd2" member_type: VOTER last_known_addr { host: "127.10.198.1" port: 38019 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e58bbdc59c354af99870745d454b2612" member_type: VOTER last_known_addr { host: "127.10.198.2" port: 34917 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f1eb71f399ba4ed0bbedf06923f39070" member_type: VOTER last_known_addr { host: "127.10.198.3" port: 41659 } health_report { overall_health: HEALTHY } } }
Sep 25 22:44:27 dist-test-slave-5vdr krb5kdc[12103](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376266, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.198.62@KRBTEST.COM for kudu/127.10.198.62@KRBTEST.COM
I20260925 22:44:27.588416 12689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:44:27.577992 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49818 (local address 127.10.198.62:42797)
0925 22:44:27.578278 (+ 286us) server_negotiation.cc:206] Beginning negotiation
0925 22:44:27.578287 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 22:44:27.578424 (+ 137us) server_negotiation.cc:407] Connection header received
0925 22:44:27.578598 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:44:27.578601 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:44:27.578650 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:44:27.578752 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:44:27.579511 (+ 759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:44:27.580182 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:44:27.581038 (+ 856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:44:27.581273 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:44:27.581390 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:44:27.584186 (+ 2796us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:44:27.584189 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:44:27.584191 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:44:27.584221 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:44:27.586231 (+ 2010us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:44:27.586840 (+ 609us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:44:27.586844 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:44:27.586846 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:44:27.586949 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:44:27.587250 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:44:27.587253 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:44:27.587255 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:44:27.587504 (+ 249us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:44:27.587610 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0925 22:44:27.588045 (+ 435us) server_negotiation.cc:299] Negotiation successful
0925 22:44:27.588269 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":106,"server-negotiator.queue_time_us":224,"thread_start_us":136,"threads_started":1}
I20260925 22:44:27.588415 12688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:44:27.577910 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.198.62:42797 (local address 127.0.0.1:49818)
0925 22:44:27.578302 (+ 392us) negotiation.cc:107] Waiting for socket to connect
0925 22:44:27.578321 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0925 22:44:27.578491 (+ 170us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:44:27.578774 (+ 283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:44:27.578777 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:44:27.579246 (+ 469us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:44:27.579351 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:44:27.579359 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:44:27.580319 (+ 960us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:44:27.580322 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:44:27.580903 (+ 581us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:44:27.580911 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:44:27.581407 (+ 496us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:44:27.581411 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:44:27.581503 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:44:27.582191 (+ 688us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:44:27.582205 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:44:27.583950 (+ 1745us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:44:27.586398 (+ 2448us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:44:27.586411 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:44:27.586417 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:44:27.586712 (+ 295us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:44:27.587051 (+ 339us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:44:27.587053 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:44:27.587054 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:44:27.587154 (+ 100us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:44:27.587639 (+ 485us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:44:27.587646 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:44:27.587774 (+ 128us) client_negotiation.cc:769] Sending connection context
0925 22:44:27.587985 (+ 211us) client_negotiation.cc:240] Negotiation successful
0925 22:44:27.588197 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":276,"mutex_wait_us":1,"thread_start_us":142,"threads_started":1}
I20260925 22:44:27.595471 12146 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.10.198.62@KRBTEST.COM'} at 127.0.0.1:49818:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260925 22:44:27.596345 12146 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 22:44:27.602352 12285 tablet_service.cc:1467] Processing CreateTablet for tablet 94b7ab3f4f614cbaa625bc74654d1b75 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=f8a7aabe0b6746deaf308883a172f0ae]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260925 22:44:27.602352 12425 tablet_service.cc:1467] Processing CreateTablet for tablet 94b7ab3f4f614cbaa625bc74654d1b75 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=f8a7aabe0b6746deaf308883a172f0ae]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260925 22:44:27.602526 12565 tablet_service.cc:1467] Processing CreateTablet for tablet 94b7ab3f4f614cbaa625bc74654d1b75 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=f8a7aabe0b6746deaf308883a172f0ae]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260925 22:44:27.602810 12425 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 94b7ab3f4f614cbaa625bc74654d1b75. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:44:27.602810 12285 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 94b7ab3f4f614cbaa625bc74654d1b75. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:44:27.602922 12565 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 94b7ab3f4f614cbaa625bc74654d1b75. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:44:27.606348 12655 tablet_bootstrap.cc:492] T 94b7ab3f4f614cbaa625bc74654d1b75 P e58bbdc59c354af99870745d454b2612: Bootstrap starting.
I20260925 22:44:27.606354 12656 tablet_bootstrap.cc:492] T 94b7ab3f4f614cbaa625bc74654d1b75 P f1eb71f399ba4ed0bbedf06923f39070: Bootstrap starting.
I20260925 22:44:27.606436 12654 tablet_bootstrap.cc:492] T 94b7ab3f4f614cbaa625bc74654d1b75 P 3065b552ce554279b34368a4fec28dd2: Bootstrap starting.
I20260925 22:44:27.607443 12654 tablet_bootstrap.cc:654] T 94b7ab3f4f614cbaa625bc74654d1b75 P 3065b552ce554279b34368a4fec28dd2: Neither blocks nor log segments found. Creating new log.
I20260925 22:44:27.607443 12655 tablet_bootstrap.cc:654] T 94b7ab3f4f614cbaa625bc74654d1b75 P e58bbdc59c354af99870745d454b2612: Neither blocks nor log segments found. Creating new log.
I20260925 22:44:27.607466 12656 tablet_bootstrap.cc:654] T 94b7ab3f4f614cbaa625bc74654d1b75 P f1eb71f399ba4ed0bbedf06923f39070: Neither blocks nor log segments found. Creating new log.
I20260925 22:44:27.609573 12656 tablet_bootstrap.cc:492] T 94b7ab3f4f614cbaa625bc74654d1b75 P f1eb71f399ba4ed0bbedf06923f39070: No bootstrap required, opened a new log
I20260925 22:44:27.609694 12656 ts_tablet_manager.cc:1397] T 94b7ab3f4f614cbaa625bc74654d1b75 P f1eb71f399ba4ed0bbedf06923f39070: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260925 22:44:27.609817 12654 tablet_bootstrap.cc:492] T 94b7ab3f4f614cbaa625bc74654d1b75 P 3065b552ce554279b34368a4fec28dd2: No bootstrap required, opened a new log
I20260925 22:44:27.609948 12654 ts_tablet_manager.cc:1397] T 94b7ab3f4f614cbaa625bc74654d1b75 P 3065b552ce554279b34368a4fec28dd2: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.002s
I20260925 22:44:27.610203 12656 raft_consensus.cc:348] T 94b7ab3f4f614cbaa625bc74654d1b75 P f1eb71f399ba4ed0bbedf06923f39070 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3065b552ce554279b34368a4fec28dd2" member_type: VOTER last_known_addr { host: "127.10.198.1" port: 38019 } } peers { permanent_uuid: "e58bbdc59c354af99870745d454b2612" member_type: VOTER last_known_addr { host: "127.10.198.2" port: 34917 } } peers { permanent_uuid: "f1eb71f399ba4ed0bbedf06923f39070" member_type: VOTER last_known_addr { host: "127.10.198.3" port: 41659 } }
I20260925 22:44:27.610357 12656 raft_consensus.cc:374] T 94b7ab3f4f614cbaa625bc74654d1b75 P f1eb71f399ba4ed0bbedf06923f39070 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:44:27.610394 12656 raft_consensus.cc:729] T 94b7ab3f4f614cbaa625bc74654d1b75 P f1eb71f399ba4ed0bbedf06923f39070 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1eb71f399ba4ed0bbedf06923f39070, State: Initialized, Role: FOLLOWER
I20260925 22:44:27.610394 12654 raft_consensus.cc:348] T 94b7ab3f4f614cbaa625bc74654d1b75 P 3065b552ce554279b34368a4fec28dd2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3065b552ce554279b34368a4fec28dd2" member_type: VOTER last_known_addr { host: "127.10.198.1" port: 38019 } } peers { permanent_uuid: "e58bbdc59c354af99870745d454b2612" member_type: VOTER last_known_addr { host: "127.10.198.2" port: 34917 } } peers { permanent_uuid: "f1eb71f399ba4ed0bbedf06923f39070" member_type: VOTER last_known_addr { host: "127.10.198.3" port: 41659 } }
I20260925 22:44:27.610526 12654 raft_consensus.cc:374] T 94b7ab3f4f614cbaa625bc74654d1b75 P 3065b552ce554279b34368a4fec28dd2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:44:27.610529 12656 consensus_queue.cc:260] T 94b7ab3f4f614cbaa625bc74654d1b75 P f1eb71f399ba4ed0bbedf06923f39070 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3065b552ce554279b34368a4fec28dd2" member_type: VOTER last_known_addr { host: "127.10.198.1" port: 38019 } } peers { permanent_uuid: "e58bbdc59c354af99870745d454b2612" member_type: VOTER last_known_addr { host: "127.10.198.2" port: 34917 } } peers { permanent_uuid: "f1eb71f399ba4ed0bbedf06923f39070" member_type: VOTER last_known_addr { host: "127.10.198.3" port: 41659 } }
I20260925 22:44:27.610571 12654 raft_consensus.cc:729] T 94b7ab3f4f614cbaa625bc74654d1b75 P 3065b552ce554279b34368a4fec28dd2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3065b552ce554279b34368a4fec28dd2, State: Initialized, Role: FOLLOWER
I20260925 22:44:27.610751 12654 consensus_queue.cc:260] T 94b7ab3f4f614cbaa625bc74654d1b75 P 3065b552ce554279b34368a4fec28dd2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3065b552ce554279b34368a4fec28dd2" member_type: VOTER last_known_addr { host: "127.10.198.1" port: 38019 } } peers { permanent_uuid: "e58bbdc59c354af99870745d454b2612" member_type: VOTER last_known_addr { host: "127.10.198.2" port: 34917 } } peers { permanent_uuid: "f1eb71f399ba4ed0bbedf06923f39070" member_type: VOTER last_known_addr { host: "127.10.198.3" port: 41659 } }
I20260925 22:44:27.610819 12656 ts_tablet_manager.cc:1428] T 94b7ab3f4f614cbaa625bc74654d1b75 P f1eb71f399ba4ed0bbedf06923f39070: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20260925 22:44:27.611032 12661 tablet_replica.cc:439] T 94b7ab3f4f614cbaa625bc74654d1b75 P 3065b552ce554279b34368a4fec28dd2: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3065b552ce554279b34368a4fec28dd2" member_type: VOTER last_known_addr { host: "127.10.198.1" port: 38019 } } peers { permanent_uuid: "e58bbdc59c354af99870745d454b2612" member_type: VOTER last_known_addr { host: "127.10.198.2" port: 34917 } } peers { permanent_uuid: "f1eb71f399ba4ed0bbedf06923f39070" member_type: VOTER last_known_addr { host: "127.10.198.3" port: 41659 } } }
I20260925 22:44:27.611080 12664 tablet_replica.cc:439] T 94b7ab3f4f614cbaa625bc74654d1b75 P f1eb71f399ba4ed0bbedf06923f39070: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3065b552ce554279b34368a4fec28dd2" member_type: VOTER last_known_addr { host: "127.10.198.1" port: 38019 } } peers { permanent_uuid: "e58bbdc59c354af99870745d454b2612" member_type: VOTER last_known_addr { host: "127.10.198.2" port: 34917 } } peers { permanent_uuid: "f1eb71f399ba4ed0bbedf06923f39070" member_type: VOTER last_known_addr { host: "127.10.198.3" port: 41659 } } }
I20260925 22:44:27.611182 12661 tablet_replica.cc:442] T 94b7ab3f4f614cbaa625bc74654d1b75 P 3065b552ce554279b34368a4fec28dd2: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 22:44:27.611204 12664 tablet_replica.cc:442] T 94b7ab3f4f614cbaa625bc74654d1b75 P f1eb71f399ba4ed0bbedf06923f39070: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 22:44:27.612763 12654 ts_tablet_manager.cc:1428] T 94b7ab3f4f614cbaa625bc74654d1b75 P 3065b552ce554279b34368a4fec28dd2: Time spent starting tablet: real 0.003s user 0.000s sys 0.001s
I20260925 22:44:27.613562 12655 tablet_bootstrap.cc:492] T 94b7ab3f4f614cbaa625bc74654d1b75 P e58bbdc59c354af99870745d454b2612: No bootstrap required, opened a new log
I20260925 22:44:27.613690 12655 ts_tablet_manager.cc:1397] T 94b7ab3f4f614cbaa625bc74654d1b75 P e58bbdc59c354af99870745d454b2612: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.002s
I20260925 22:44:27.614158 12655 raft_consensus.cc:348] T 94b7ab3f4f614cbaa625bc74654d1b75 P e58bbdc59c354af99870745d454b2612 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3065b552ce554279b34368a4fec28dd2" member_type: VOTER last_known_addr { host: "127.10.198.1" port: 38019 } } peers { permanent_uuid: "e58bbdc59c354af99870745d454b2612" member_type: VOTER last_known_addr { host: "127.10.198.2" port: 34917 } } peers { permanent_uuid: "f1eb71f399ba4ed0bbedf06923f39070" member_type: VOTER last_known_addr { host: "127.10.198.3" port: 41659 } }
I20260925 22:44:27.614289 12655 raft_consensus.cc:374] T 94b7ab3f4f614cbaa625bc74654d1b75 P e58bbdc59c354af99870745d454b2612 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:44:27.614331 12655 raft_consensus.cc:729] T 94b7ab3f4f614cbaa625bc74654d1b75 P e58bbdc59c354af99870745d454b2612 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e58bbdc59c354af99870745d454b2612, State: Initialized, Role: FOLLOWER
I20260925 22:44:27.614471 12655 consensus_queue.cc:260] T 94b7ab3f4f614cbaa625bc74654d1b75 P e58bbdc59c354af99870745d454b2612 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3065b552ce554279b34368a4fec28dd2" member_type: VOTER last_known_addr { host: "127.10.198.1" port: 38019 } } peers { permanent_uuid: "e58bbdc59c354af99870745d454b2612" member_type: VOTER last_known_addr { host: "127.10.198.2" port: 34917 } } peers { permanent_uuid: "f1eb71f399ba4ed0bbedf06923f39070" member_type: VOTER last_known_addr { host: "127.10.198.3" port: 41659 } }
I20260925 22:44:27.614785 12655 ts_tablet_manager.cc:1428] T 94b7ab3f4f614cbaa625bc74654d1b75 P e58bbdc59c354af99870745d454b2612: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260925 22:44:27.614795 12662 tablet_replica.cc:439] T 94b7ab3f4f614cbaa625bc74654d1b75 P e58bbdc59c354af99870745d454b2612: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3065b552ce554279b34368a4fec28dd2" member_type: VOTER last_known_addr { host: "127.10.198.1" port: 38019 } } peers { permanent_uuid: "e58bbdc59c354af99870745d454b2612" member_type: VOTER last_known_addr { host: "127.10.198.2" port: 34917 } } peers { permanent_uuid: "f1eb71f399ba4ed0bbedf06923f39070" member_type: VOTER last_known_addr { host: "127.10.198.3" port: 41659 } } }
I20260925 22:44:27.615355 12662 tablet_replica.cc:442] T 94b7ab3f4f614cbaa625bc74654d1b75 P e58bbdc59c354af99870745d454b2612: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 22:44:27.703382 12662 raft_consensus.cc:482] T 94b7ab3f4f614cbaa625bc74654d1b75 P e58bbdc59c354af99870745d454b2612 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 22:44:27.703552 12662 raft_consensus.cc:504] T 94b7ab3f4f614cbaa625bc74654d1b75 P e58bbdc59c354af99870745d454b2612 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3065b552ce554279b34368a4fec28dd2" member_type: VOTER last_known_addr { host: "127.10.198.1" port: 38019 } } peers { permanent_uuid: "e58bbdc59c354af99870745d454b2612" member_type: VOTER last_known_addr { host: "127.10.198.2" port: 34917 } } peers { permanent_uuid: "f1eb71f399ba4ed0bbedf06923f39070" member_type: VOTER last_known_addr { host: "127.10.198.3" port: 41659 } }
I20260925 22:44:27.704715 12662 leader_election.cc:288] T 94b7ab3f4f614cbaa625bc74654d1b75 P e58bbdc59c354af99870745d454b2612 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3065b552ce554279b34368a4fec28dd2 (127.10.198.1:38019), f1eb71f399ba4ed0bbedf06923f39070 (127.10.198.3:41659)
I20260925 22:44:27.708973 12652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:44:27.705094 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.198.2:45857 (local address 127.10.198.3:41659)
0925 22:44:27.705287 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0925 22:44:27.705293 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 22:44:27.705678 (+ 385us) server_negotiation.cc:407] Connection header received
0925 22:44:27.705841 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:44:27.705844 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:44:27.705899 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:44:27.705985 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:44:27.706239 (+ 254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:44:27.706734 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:44:27.707863 (+ 1129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:44:27.708329 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:44:27.708475 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:44:27.708536 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0925 22:44:27.708754 (+ 218us) server_negotiation.cc:299] Negotiation successful
0925 22:44:27.708832 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":101}
I20260925 22:44:27.708973 12650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:44:27.705232 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.198.2:48175 (local address 127.10.198.1:38019)
0925 22:44:27.705370 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0925 22:44:27.705375 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:44:27.705571 (+ 196us) server_negotiation.cc:407] Connection header received
0925 22:44:27.705745 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:44:27.705748 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:44:27.705799 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:44:27.705883 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:44:27.706155 (+ 272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:44:27.706662 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:44:27.707834 (+ 1172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:44:27.708300 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:44:27.708444 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:44:27.708501 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0925 22:44:27.708739 (+ 238us) server_negotiation.cc:299] Negotiation successful
0925 22:44:27.708828 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260925 22:44:27.709007 12694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:44:27.705004 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.198.3:41659 (local address 127.10.198.2:45857)
0925 22:44:27.705574 (+ 570us) negotiation.cc:107] Waiting for socket to connect
0925 22:44:27.705587 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 22:44:27.705735 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:44:27.705995 (+ 260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:44:27.705998 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:44:27.706013 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:44:27.706132 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:44:27.706137 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:44:27.706844 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:44:27.706849 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:44:27.707705 (+ 856us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:44:27.707716 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:44:27.708464 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:44:27.708468 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:44:27.708619 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:44:27.708637 (+ 18us) client_negotiation.cc:769] Sending connection context
0925 22:44:27.708735 (+ 98us) client_negotiation.cc:240] Negotiation successful
0925 22:44:27.708832 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":508,"mutex_wait_us":176,"thread_start_us":89,"threads_started":1}
I20260925 22:44:27.709015 12693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:44:27.705004 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.198.1:38019 (local address 127.10.198.2:48175)
0925 22:44:27.705465 (+ 461us) negotiation.cc:107] Waiting for socket to connect
0925 22:44:27.705480 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 22:44:27.705626 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:44:27.705899 (+ 273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:44:27.705902 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:44:27.705918 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:44:27.706045 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:44:27.706051 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:44:27.706783 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:44:27.706786 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:44:27.707708 (+ 922us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:44:27.707718 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:44:27.708445 (+ 727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:44:27.708449 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:44:27.708558 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:44:27.708624 (+ 66us) client_negotiation.cc:769] Sending connection context
0925 22:44:27.708719 (+ 95us) client_negotiation.cc:240] Negotiation successful
0925 22:44:27.708828 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":381,"mutex_wait_us":47,"thread_start_us":136,"threads_started":1}
I20260925 22:44:27.709755 12585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "94b7ab3f4f614cbaa625bc74654d1b75" candidate_uuid: "e58bbdc59c354af99870745d454b2612" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1eb71f399ba4ed0bbedf06923f39070" is_pre_election: true
I20260925 22:44:27.709908 12305 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "94b7ab3f4f614cbaa625bc74654d1b75" candidate_uuid: "e58bbdc59c354af99870745d454b2612" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3065b552ce554279b34368a4fec28dd2" is_pre_election: true
I20260925 22:44:27.710023 12585 raft_consensus.cc:2454] T 94b7ab3f4f614cbaa625bc74654d1b75 P f1eb71f399ba4ed0bbedf06923f39070 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e58bbdc59c354af99870745d454b2612 in term 0.
I20260925 22:44:27.710076 12305 raft_consensus.cc:2454] T 94b7ab3f4f614cbaa625bc74654d1b75 P 3065b552ce554279b34368a4fec28dd2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e58bbdc59c354af99870745d454b2612 in term 0.
I20260925 22:44:27.710462 12377 leader_election.cc:302] T 94b7ab3f4f614cbaa625bc74654d1b75 P e58bbdc59c354af99870745d454b2612 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3065b552ce554279b34368a4fec28dd2, e58bbdc59c354af99870745d454b2612; no voters:
I20260925 22:44:27.710779 12662 raft_consensus.cc:2781] T 94b7ab3f4f614cbaa625bc74654d1b75 P e58bbdc59c354af99870745d454b2612 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 22:44:27.710865 12662 raft_consensus.cc:482] T 94b7ab3f4f614cbaa625bc74654d1b75 P e58bbdc59c354af99870745d454b2612 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 22:44:27.710918 12662 raft_consensus.cc:3037] T 94b7ab3f4f614cbaa625bc74654d1b75 P e58bbdc59c354af99870745d454b2612 [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:44:27.712069 12662 raft_consensus.cc:504] T 94b7ab3f4f614cbaa625bc74654d1b75 P e58bbdc59c354af99870745d454b2612 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3065b552ce554279b34368a4fec28dd2" member_type: VOTER last_known_addr { host: "127.10.198.1" port: 38019 } } peers { permanent_uuid: "e58bbdc59c354af99870745d454b2612" member_type: VOTER last_known_addr { host: "127.10.198.2" port: 34917 } } peers { permanent_uuid: "f1eb71f399ba4ed0bbedf06923f39070" member_type: VOTER last_known_addr { host: "127.10.198.3" port: 41659 } }
I20260925 22:44:27.712481 12662 leader_election.cc:288] T 94b7ab3f4f614cbaa625bc74654d1b75 P e58bbdc59c354af99870745d454b2612 [CANDIDATE]: Term 1 election: Requested vote from peers 3065b552ce554279b34368a4fec28dd2 (127.10.198.1:38019), f1eb71f399ba4ed0bbedf06923f39070 (127.10.198.3:41659)
I20260925 22:44:27.712965 12585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "94b7ab3f4f614cbaa625bc74654d1b75" candidate_uuid: "e58bbdc59c354af99870745d454b2612" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1eb71f399ba4ed0bbedf06923f39070"
I20260925 22:44:27.713120 12585 raft_consensus.cc:3037] T 94b7ab3f4f614cbaa625bc74654d1b75 P f1eb71f399ba4ed0bbedf06923f39070 [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:44:27.713241 12305 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "94b7ab3f4f614cbaa625bc74654d1b75" candidate_uuid: "e58bbdc59c354af99870745d454b2612" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3065b552ce554279b34368a4fec28dd2"
I20260925 22:44:27.713399 12305 raft_consensus.cc:3037] T 94b7ab3f4f614cbaa625bc74654d1b75 P 3065b552ce554279b34368a4fec28dd2 [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:44:27.714218 12585 raft_consensus.cc:2454] T 94b7ab3f4f614cbaa625bc74654d1b75 P f1eb71f399ba4ed0bbedf06923f39070 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e58bbdc59c354af99870745d454b2612 in term 1.
I20260925 22:44:27.714357 12305 raft_consensus.cc:2454] T 94b7ab3f4f614cbaa625bc74654d1b75 P 3065b552ce554279b34368a4fec28dd2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e58bbdc59c354af99870745d454b2612 in term 1.
I20260925 22:44:27.714629 12376 leader_election.cc:302] T 94b7ab3f4f614cbaa625bc74654d1b75 P e58bbdc59c354af99870745d454b2612 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e58bbdc59c354af99870745d454b2612, f1eb71f399ba4ed0bbedf06923f39070; no voters:
I20260925 22:44:27.714826 12662 raft_consensus.cc:2781] T 94b7ab3f4f614cbaa625bc74654d1b75 P e58bbdc59c354af99870745d454b2612 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 22:44:27.715044 12662 raft_consensus.cc:686] T 94b7ab3f4f614cbaa625bc74654d1b75 P e58bbdc59c354af99870745d454b2612 [term 1 LEADER]: Becoming Leader. State: Replica: e58bbdc59c354af99870745d454b2612, State: Running, Role: LEADER
I20260925 22:44:27.715328 12662 consensus_queue.cc:237] T 94b7ab3f4f614cbaa625bc74654d1b75 P e58bbdc59c354af99870745d454b2612 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3065b552ce554279b34368a4fec28dd2" member_type: VOTER last_known_addr { host: "127.10.198.1" port: 38019 } } peers { permanent_uuid: "e58bbdc59c354af99870745d454b2612" member_type: VOTER last_known_addr { host: "127.10.198.2" port: 34917 } } peers { permanent_uuid: "f1eb71f399ba4ed0bbedf06923f39070" member_type: VOTER last_known_addr { host: "127.10.198.3" port: 41659 } }
I20260925 22:44:27.716774 12695 tablet_replica.cc:439] T 94b7ab3f4f614cbaa625bc74654d1b75 P e58bbdc59c354af99870745d454b2612: TxnStatusTablet state changed. Reason: New leader e58bbdc59c354af99870745d454b2612. Latest consensus state: current_term: 1 leader_uuid: "e58bbdc59c354af99870745d454b2612" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3065b552ce554279b34368a4fec28dd2" member_type: VOTER last_known_addr { host: "127.10.198.1" port: 38019 } } peers { permanent_uuid: "e58bbdc59c354af99870745d454b2612" member_type: VOTER last_known_addr { host: "127.10.198.2" port: 34917 } } peers { permanent_uuid: "f1eb71f399ba4ed0bbedf06923f39070" member_type: VOTER last_known_addr { host: "127.10.198.3" port: 41659 } } }
I20260925 22:44:27.716957 12695 tablet_replica.cc:442] T 94b7ab3f4f614cbaa625bc74654d1b75 P e58bbdc59c354af99870745d454b2612: This TxnStatusTablet replica's current role is: LEADER
I20260925 22:44:27.717289 12697 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260925 22:44:27.717476 12697 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260925 22:44:27.718144 12146 catalog_manager.cc:5495] T 94b7ab3f4f614cbaa625bc74654d1b75 P e58bbdc59c354af99870745d454b2612 reported cstate change: term changed from 0 to 1, leader changed from <none> to e58bbdc59c354af99870745d454b2612 (127.10.198.2). New cstate: current_term: 1 leader_uuid: "e58bbdc59c354af99870745d454b2612" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3065b552ce554279b34368a4fec28dd2" member_type: VOTER last_known_addr { host: "127.10.198.1" port: 38019 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e58bbdc59c354af99870745d454b2612" member_type: VOTER last_known_addr { host: "127.10.198.2" port: 34917 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f1eb71f399ba4ed0bbedf06923f39070" member_type: VOTER last_known_addr { host: "127.10.198.3" port: 41659 } health_report { overall_health: UNKNOWN } } }
I20260925 22:44:27.745332 12653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:44:27.740816 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59030 (local address 127.10.198.2:34917)
0925 22:44:27.740951 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0925 22:44:27.740955 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:44:27.741055 (+ 100us) server_negotiation.cc:407] Connection header received
0925 22:44:27.741302 (+ 247us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:44:27.741304 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:44:27.741362 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:44:27.741456 (+ 94us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 22:44:27.741757 (+ 301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:44:27.742252 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:44:27.743622 (+ 1370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:44:27.743935 (+ 313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:44:27.744036 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:44:27.744408 (+ 372us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 22:44:27.744914 (+ 506us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 22:44:27.745009 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0925 22:44:27.745136 (+ 127us) server_negotiation.cc:299] Negotiation successful
0925 22:44:27.745191 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260925 22:44:27.745393 12688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:44:27.740702 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.198.2:34917 (local address 127.0.0.1:59030)
0925 22:44:27.740987 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0925 22:44:27.741004 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 22:44:27.741138 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:44:27.741509 (+ 371us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:44:27.741513 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:44:27.741522 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 22:44:27.741645 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:44:27.741652 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:44:27.742389 (+ 737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:44:27.742393 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:44:27.743425 (+ 1032us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:44:27.743440 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:44:27.744068 (+ 628us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:44:27.744072 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:44:27.744183 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:44:27.744255 (+ 72us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 22:44:27.745017 (+ 762us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 22:44:27.745038 (+ 21us) client_negotiation.cc:769] Sending connection context
0925 22:44:27.745117 (+ 79us) client_negotiation.cc:240] Negotiation successful
0925 22:44:27.745236 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":171}
I20260925 22:44:27.749179 12305 raft_consensus.cc:1261] T 94b7ab3f4f614cbaa625bc74654d1b75 P 3065b552ce554279b34368a4fec28dd2 [term 1 FOLLOWER]: Refusing update from remote peer e58bbdc59c354af99870745d454b2612: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 22:44:27.749359 12585 raft_consensus.cc:1261] T 94b7ab3f4f614cbaa625bc74654d1b75 P f1eb71f399ba4ed0bbedf06923f39070 [term 1 FOLLOWER]: Refusing update from remote peer e58bbdc59c354af99870745d454b2612: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 22:44:27.749788 12662 consensus_queue.cc:1035] T 94b7ab3f4f614cbaa625bc74654d1b75 P e58bbdc59c354af99870745d454b2612 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3065b552ce554279b34368a4fec28dd2" member_type: VOTER last_known_addr { host: "127.10.198.1" port: 38019 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 22:44:27.750100 12695 consensus_queue.cc:1035] T 94b7ab3f4f614cbaa625bc74654d1b75 P e58bbdc59c354af99870745d454b2612 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f1eb71f399ba4ed0bbedf06923f39070" member_type: VOTER last_known_addr { host: "127.10.198.3" port: 41659 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 22:44:27.751586 12661 tablet_replica.cc:439] T 94b7ab3f4f614cbaa625bc74654d1b75 P 3065b552ce554279b34368a4fec28dd2: TxnStatusTablet state changed. Reason: New leader e58bbdc59c354af99870745d454b2612. Latest consensus state: current_term: 1 leader_uuid: "e58bbdc59c354af99870745d454b2612" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3065b552ce554279b34368a4fec28dd2" member_type: VOTER last_known_addr { host: "127.10.198.1" port: 38019 } } peers { permanent_uuid: "e58bbdc59c354af99870745d454b2612" member_type: VOTER last_known_addr { host: "127.10.198.2" port: 34917 } } peers { permanent_uuid: "f1eb71f399ba4ed0bbedf06923f39070" member_type: VOTER last_known_addr { host: "127.10.198.3" port: 41659 } } }
I20260925 22:44:27.751740 12661 tablet_replica.cc:442] T 94b7ab3f4f614cbaa625bc74654d1b75 P 3065b552ce554279b34368a4fec28dd2: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 22:44:27.751866 12664 tablet_replica.cc:439] T 94b7ab3f4f614cbaa625bc74654d1b75 P f1eb71f399ba4ed0bbedf06923f39070: TxnStatusTablet state changed. Reason: New leader e58bbdc59c354af99870745d454b2612. Latest consensus state: current_term: 1 leader_uuid: "e58bbdc59c354af99870745d454b2612" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3065b552ce554279b34368a4fec28dd2" member_type: VOTER last_known_addr { host: "127.10.198.1" port: 38019 } } peers { permanent_uuid: "e58bbdc59c354af99870745d454b2612" member_type: VOTER last_known_addr { host: "127.10.198.2" port: 34917 } } peers { permanent_uuid: "f1eb71f399ba4ed0bbedf06923f39070" member_type: VOTER last_known_addr { host: "127.10.198.3" port: 41659 } } }
I20260925 22:44:27.751992 12664 tablet_replica.cc:442] T 94b7ab3f4f614cbaa625bc74654d1b75 P f1eb71f399ba4ed0bbedf06923f39070: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 22:44:27.755955 12662 tablet_replica.cc:439] T 94b7ab3f4f614cbaa625bc74654d1b75 P e58bbdc59c354af99870745d454b2612: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "e58bbdc59c354af99870745d454b2612" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3065b552ce554279b34368a4fec28dd2" member_type: VOTER last_known_addr { host: "127.10.198.1" port: 38019 } } peers { permanent_uuid: "e58bbdc59c354af99870745d454b2612" member_type: VOTER last_known_addr { host: "127.10.198.2" port: 34917 } } peers { permanent_uuid: "f1eb71f399ba4ed0bbedf06923f39070" member_type: VOTER last_known_addr { host: "127.10.198.3" port: 41659 } } }
I20260925 22:44:27.756110 12662 tablet_replica.cc:442] T 94b7ab3f4f614cbaa625bc74654d1b75 P e58bbdc59c354af99870745d454b2612: This TxnStatusTablet replica's current role is: LEADER
I20260925 22:44:27.756625 12661 tablet_replica.cc:439] T 94b7ab3f4f614cbaa625bc74654d1b75 P 3065b552ce554279b34368a4fec28dd2: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "e58bbdc59c354af99870745d454b2612" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3065b552ce554279b34368a4fec28dd2" member_type: VOTER last_known_addr { host: "127.10.198.1" port: 38019 } } peers { permanent_uuid: "e58bbdc59c354af99870745d454b2612" member_type: VOTER last_known_addr { host: "127.10.198.2" port: 34917 } } peers { permanent_uuid: "f1eb71f399ba4ed0bbedf06923f39070" member_type: VOTER last_known_addr { host: "127.10.198.3" port: 41659 } } }
I20260925 22:44:27.756768 12661 tablet_replica.cc:442] T 94b7ab3f4f614cbaa625bc74654d1b75 P 3065b552ce554279b34368a4fec28dd2: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 22:44:27.757756 12695 tablet_replica.cc:439] T 94b7ab3f4f614cbaa625bc74654d1b75 P e58bbdc59c354af99870745d454b2612: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "e58bbdc59c354af99870745d454b2612" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3065b552ce554279b34368a4fec28dd2" member_type: VOTER last_known_addr { host: "127.10.198.1" port: 38019 } } peers { permanent_uuid: "e58bbdc59c354af99870745d454b2612" member_type: VOTER last_known_addr { host: "127.10.198.2" port: 34917 } } peers { permanent_uuid: "f1eb71f399ba4ed0bbedf06923f39070" member_type: VOTER last_known_addr { host: "127.10.198.3" port: 41659 } } }
I20260925 22:44:27.757889 12695 tablet_replica.cc:442] T 94b7ab3f4f614cbaa625bc74654d1b75 P e58bbdc59c354af99870745d454b2612: This TxnStatusTablet replica's current role is: LEADER
I20260925 22:44:27.760957 12664 tablet_replica.cc:439] T 94b7ab3f4f614cbaa625bc74654d1b75 P f1eb71f399ba4ed0bbedf06923f39070: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "e58bbdc59c354af99870745d454b2612" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3065b552ce554279b34368a4fec28dd2" member_type: VOTER last_known_addr { host: "127.10.198.1" port: 38019 } } peers { permanent_uuid: "e58bbdc59c354af99870745d454b2612" member_type: VOTER last_known_addr { host: "127.10.198.2" port: 34917 } } peers { permanent_uuid: "f1eb71f399ba4ed0bbedf06923f39070" member_type: VOTER last_known_addr { host: "127.10.198.3" port: 41659 } } }
I20260925 22:44:27.761137 12664 tablet_replica.cc:442] T 94b7ab3f4f614cbaa625bc74654d1b75 P f1eb71f399ba4ed0bbedf06923f39070: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 22:44:27.774646 12653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:44:27.769874 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.198.3:58159 (local address 127.10.198.2:34917)
0925 22:44:27.770043 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0925 22:44:27.770048 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:44:27.770230 (+ 182us) server_negotiation.cc:407] Connection header received
0925 22:44:27.770429 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:44:27.770432 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:44:27.770482 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:44:27.770542 (+ 60us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 22:44:27.770909 (+ 367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:44:27.771696 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:44:27.773108 (+ 1412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:44:27.773460 (+ 352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:44:27.773546 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:44:27.774013 (+ 467us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 22:44:27.774160 (+ 147us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 22:44:27.774254 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0925 22:44:27.774420 (+ 166us) server_negotiation.cc:299] Negotiation successful
0925 22:44:27.774484 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260925 22:44:27.774669 12705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:44:27.769814 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.198.2:34917 (local address 127.10.198.3:58159)
0925 22:44:27.770138 (+ 324us) negotiation.cc:107] Waiting for socket to connect
0925 22:44:27.770153 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 22:44:27.770315 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:44:27.770683 (+ 368us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:44:27.770686 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:44:27.770695 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 22:44:27.770800 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:44:27.770805 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:44:27.771896 (+ 1091us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:44:27.771899 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:44:27.772924 (+ 1025us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:44:27.772936 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:44:27.773674 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:44:27.773678 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:44:27.773773 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:44:27.773838 (+ 65us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 22:44:27.774304 (+ 466us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 22:44:27.774312 (+ 8us) client_negotiation.cc:769] Sending connection context
0925 22:44:27.774386 (+ 74us) client_negotiation.cc:240] Negotiation successful
0925 22:44:27.774504 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":211,"thread_start_us":117,"threads_started":1}
I20260925 22:44:27.782467 12445 raft_consensus.cc:1261] T 9a096b4ae2d944138a7d928c3fd1bc71 P e58bbdc59c354af99870745d454b2612 [term 1 FOLLOWER]: Refusing update from remote peer f1eb71f399ba4ed0bbedf06923f39070: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 22:44:27.782467 12305 raft_consensus.cc:1261] T 9a096b4ae2d944138a7d928c3fd1bc71 P 3065b552ce554279b34368a4fec28dd2 [term 1 FOLLOWER]: Refusing update from remote peer f1eb71f399ba4ed0bbedf06923f39070: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 22:44:27.782882 12660 consensus_queue.cc:1035] T 9a096b4ae2d944138a7d928c3fd1bc71 P f1eb71f399ba4ed0bbedf06923f39070 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3065b552ce554279b34368a4fec28dd2" member_type: VOTER last_known_addr { host: "127.10.198.1" port: 38019 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 22:44:27.783311 12660 consensus_queue.cc:1035] T 9a096b4ae2d944138a7d928c3fd1bc71 P f1eb71f399ba4ed0bbedf06923f39070 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e58bbdc59c354af99870745d454b2612" member_type: VOTER last_known_addr { host: "127.10.198.2" port: 34917 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 22:44:27.787787 12670 mvcc.cc:205] Tried to move back new op lower bound from 7333381192833081344 to 7333381191875719168. Current Snapshot: MvccSnapshot[applied={T|T < 7333381192833081344}]
I20260925 22:44:27.808678 12710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:44:27.804635 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.198.3:41659 (local address 127.10.198.2:55833)
0925 22:44:27.805127 (+ 492us) negotiation.cc:107] Waiting for socket to connect
0925 22:44:27.805141 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 22:44:27.805289 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:44:27.805615 (+ 326us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:44:27.805618 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:44:27.805627 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 22:44:27.805751 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:44:27.805756 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:44:27.806522 (+ 766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:44:27.806526 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:44:27.807321 (+ 795us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:44:27.807331 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:44:27.807810 (+ 479us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:44:27.807814 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:44:27.807958 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:44:27.808013 (+ 55us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 22:44:27.808364 (+ 351us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 22:44:27.808372 (+ 8us) client_negotiation.cc:769] Sending connection context
0925 22:44:27.808437 (+ 65us) client_negotiation.cc:240] Negotiation successful
0925 22:44:27.808533 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":372,"thread_start_us":137,"threads_started":1}
I20260925 22:44:27.808662 12652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:44:27.804773 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.198.2:55833 (local address 127.10.198.3:41659)
0925 22:44:27.804921 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0925 22:44:27.804926 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:44:27.805392 (+ 466us) server_negotiation.cc:407] Connection header received
0925 22:44:27.805455 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:44:27.805457 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:44:27.805503 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:44:27.805589 (+ 86us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 22:44:27.805861 (+ 272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:44:27.806391 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:44:27.807449 (+ 1058us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:44:27.807690 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:44:27.807809 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:44:27.808164 (+ 355us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 22:44:27.808267 (+ 103us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 22:44:27.808355 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0925 22:44:27.808447 (+ 92us) server_negotiation.cc:299] Negotiation successful
0925 22:44:27.808509 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260925 22:44:27.839277 12147 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:49804:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 22:44:27.839445 12147 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:49804:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 22:44:27.841969 12147 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d5451cd6c49743beb1c5b14b5163a164: Sending DeleteTablet for 3 replicas of tablet 9a096b4ae2d944138a7d928c3fd1bc71
I20260925 22:44:27.842764 12425 tablet_service.cc:1514] Processing DeleteTablet for tablet 9a096b4ae2d944138a7d928c3fd1bc71 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 22:44:27 UTC) from {username='kudu', principal='kudu/127.10.198.62@KRBTEST.COM'} at 127.0.0.1:59018
I20260925 22:44:27.842764 12285 tablet_service.cc:1514] Processing DeleteTablet for tablet 9a096b4ae2d944138a7d928c3fd1bc71 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 22:44:27 UTC) from {username='kudu', principal='kudu/127.10.198.62@KRBTEST.COM'} at 127.0.0.1:36946
I20260925 22:44:27.842896 12565 tablet_service.cc:1514] Processing DeleteTablet for tablet 9a096b4ae2d944138a7d928c3fd1bc71 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 22:44:27 UTC) from {username='kudu', principal='kudu/127.10.198.62@KRBTEST.COM'} at 127.0.0.1:47012
I20260925 22:44:27.842969 12676 tablet_replica.cc:330] T 9a096b4ae2d944138a7d928c3fd1bc71 P e58bbdc59c354af99870745d454b2612: stopping tablet replica
I20260925 22:44:27.842969 12678 tablet_replica.cc:330] T 9a096b4ae2d944138a7d928c3fd1bc71 P 3065b552ce554279b34368a4fec28dd2: stopping tablet replica
I20260925 22:44:27.843091 12676 raft_consensus.cc:2229] T 9a096b4ae2d944138a7d928c3fd1bc71 P e58bbdc59c354af99870745d454b2612 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 22:44:27.843091 12678 raft_consensus.cc:2229] T 9a096b4ae2d944138a7d928c3fd1bc71 P 3065b552ce554279b34368a4fec28dd2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 22:44:27.843220 12678 raft_consensus.cc:2258] T 9a096b4ae2d944138a7d928c3fd1bc71 P 3065b552ce554279b34368a4fec28dd2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 22:44:27.843220 12676 raft_consensus.cc:2258] T 9a096b4ae2d944138a7d928c3fd1bc71 P e58bbdc59c354af99870745d454b2612 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 22:44:27.843631 12677 tablet_replica.cc:330] T 9a096b4ae2d944138a7d928c3fd1bc71 P f1eb71f399ba4ed0bbedf06923f39070: stopping tablet replica
I20260925 22:44:27.843791 12677 raft_consensus.cc:2229] T 9a096b4ae2d944138a7d928c3fd1bc71 P f1eb71f399ba4ed0bbedf06923f39070 [term 1 LEADER]: Raft consensus shutting down.
I20260925 22:44:27.844044 12677 raft_consensus.cc:2258] T 9a096b4ae2d944138a7d928c3fd1bc71 P f1eb71f399ba4ed0bbedf06923f39070 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 22:44:27.844415 12678 ts_tablet_manager.cc:1905] T 9a096b4ae2d944138a7d928c3fd1bc71 P 3065b552ce554279b34368a4fec28dd2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 22:44:27.847482 12678 ts_tablet_manager.cc:1918] T 9a096b4ae2d944138a7d928c3fd1bc71 P 3065b552ce554279b34368a4fec28dd2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 22:44:27.847605 12678 log.cc:1196] T 9a096b4ae2d944138a7d928c3fd1bc71 P 3065b552ce554279b34368a4fec28dd2: Deleting WAL directory at /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790376259650642-11032-0/minicluster-data/ts-0/wal/wals/9a096b4ae2d944138a7d928c3fd1bc71
I20260925 22:44:27.847926 12678 ts_tablet_manager.cc:1939] T 9a096b4ae2d944138a7d928c3fd1bc71 P 3065b552ce554279b34368a4fec28dd2: Deleting consensus metadata
I20260925 22:44:27.849052 12133 catalog_manager.cc:4841] TS 3065b552ce554279b34368a4fec28dd2 (127.10.198.1:38019): tablet 9a096b4ae2d944138a7d928c3fd1bc71 (table test-table [id=200353dfa3904226924ce522d7a61ab9]) successfully deleted
I20260925 22:44:27.849275 12677 ts_tablet_manager.cc:1905] T 9a096b4ae2d944138a7d928c3fd1bc71 P f1eb71f399ba4ed0bbedf06923f39070: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 22:44:27.849869 12676 ts_tablet_manager.cc:1905] T 9a096b4ae2d944138a7d928c3fd1bc71 P e58bbdc59c354af99870745d454b2612: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 22:44:27.852720 12677 ts_tablet_manager.cc:1918] T 9a096b4ae2d944138a7d928c3fd1bc71 P f1eb71f399ba4ed0bbedf06923f39070: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 22:44:27.852816 12677 log.cc:1196] T 9a096b4ae2d944138a7d928c3fd1bc71 P f1eb71f399ba4ed0bbedf06923f39070: Deleting WAL directory at /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790376259650642-11032-0/minicluster-data/ts-2/wal/wals/9a096b4ae2d944138a7d928c3fd1bc71
I20260925 22:44:27.852873 12676 ts_tablet_manager.cc:1918] T 9a096b4ae2d944138a7d928c3fd1bc71 P e58bbdc59c354af99870745d454b2612: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 22:44:27.852960 12676 log.cc:1196] T 9a096b4ae2d944138a7d928c3fd1bc71 P e58bbdc59c354af99870745d454b2612: Deleting WAL directory at /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790376259650642-11032-0/minicluster-data/ts-1/wal/wals/9a096b4ae2d944138a7d928c3fd1bc71
I20260925 22:44:27.853072 12677 ts_tablet_manager.cc:1939] T 9a096b4ae2d944138a7d928c3fd1bc71 P f1eb71f399ba4ed0bbedf06923f39070: Deleting consensus metadata
I20260925 22:44:27.853392 12676 ts_tablet_manager.cc:1939] T 9a096b4ae2d944138a7d928c3fd1bc71 P e58bbdc59c354af99870745d454b2612: Deleting consensus metadata
I20260925 22:44:27.854287 12134 catalog_manager.cc:4841] TS e58bbdc59c354af99870745d454b2612 (127.10.198.2:34917): tablet 9a096b4ae2d944138a7d928c3fd1bc71 (table test-table [id=200353dfa3904226924ce522d7a61ab9]) successfully deleted
I20260925 22:44:27.854636 12132 catalog_manager.cc:4841] TS f1eb71f399ba4ed0bbedf06923f39070 (127.10.198.3:41659): tablet 9a096b4ae2d944138a7d928c3fd1bc71 (table test-table [id=200353dfa3904226924ce522d7a61ab9]) successfully deleted
Sep 25 22:44:27 dist-test-slave-5vdr krb5kdc[12103](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376267, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.10.198.1@KRBTEST.COM
I20260925 22:44:27.861582 12650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:44:27.848803 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36958 (local address 127.10.198.1:38019)
0925 22:44:27.848924 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0925 22:44:27.848929 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:44:27.848947 (+ 18us) server_negotiation.cc:407] Connection header received
0925 22:44:27.848993 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:44:27.848996 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:44:27.849045 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:44:27.849125 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:44:27.850291 (+ 1166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:44:27.851036 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:44:27.852365 (+ 1329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:44:27.852744 (+ 379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:44:27.852837 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:44:27.857511 (+ 4674us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:44:27.857535 (+ 24us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:44:27.857550 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:44:27.857587 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:44:27.859475 (+ 1888us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:44:27.860111 (+ 636us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:44:27.860117 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:44:27.860118 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:44:27.860210 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:44:27.860534 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:44:27.860538 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:44:27.860539 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:44:27.860821 (+ 282us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:44:27.861083 (+ 262us) server_negotiation.cc:1035] Waiting for connection context
0925 22:44:27.861260 (+ 177us) server_negotiation.cc:299] Negotiation successful
0925 22:44:27.861401 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
W20260925 22:44:27.862190 12345 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:36958
I20260925 22:44:27.874446 12689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:44:27.864457 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49822 (local address 127.10.198.62:42797)
0925 22:44:27.864598 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0925 22:44:27.864604 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 22:44:27.864705 (+ 101us) server_negotiation.cc:407] Connection header received
0925 22:44:27.864844 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:44:27.864847 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:44:27.864895 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:44:27.864972 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:44:27.865993 (+ 1021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:44:27.866636 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:44:27.867982 (+ 1346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:44:27.868267 (+ 285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:44:27.868366 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:44:27.870251 (+ 1885us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:44:27.870255 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:44:27.870257 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:44:27.870292 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:44:27.872283 (+ 1991us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:44:27.872889 (+ 606us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:44:27.872892 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:44:27.872893 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:44:27.872987 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:44:27.873373 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:44:27.873377 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:44:27.873378 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:44:27.873652 (+ 274us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:44:27.873780 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0925 22:44:27.874124 (+ 344us) server_negotiation.cc:299] Negotiation successful
0925 22:44:27.874257 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
W20260925 22:44:27.874887 12147 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:49822
I20260925 22:44:27.876596 11032 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrWXm8X/build/debug/bin/kudu with pid 12220
I20260925 22:44:27.884514 11032 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrWXm8X/build/debug/bin/kudu with pid 12360
I20260925 22:44:27.892019 11032 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrWXm8X/build/debug/bin/kudu with pid 12500
I20260925 22:44:27.899097 11032 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrWXm8X/build/debug/bin/kudu with pid 12116
2026-09-25T22:44:27Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (3085 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790376259650642-11032-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 22:44:27 dist-test-slave-5vdr krb5kdc[12718](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 22:44:27 dist-test-slave-5vdr krb5kdc[12718](info): set up 2 sockets
Sep 25 22:44:27 dist-test-slave-5vdr krb5kdc[12718](info): commencing operation
krb5kdc: starting...
W20260925 22:44:28.955825 11032 mini_kdc.cc:121] Time spent starting KDC: real 1.033s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790376259650642-11032-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790376259650642-11032-0/krb5kdc/test-user.keytab.
Sep 25 22:44:29 dist-test-slave-5vdr krb5kdc[12718](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376269, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T22:44:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T22:44:29Z Disabled control of system clock
WARNING: no policy specified for kudu/127.10.198.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.198.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.198.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790376259650642-11032-0/krb5kdc/kudu_127.10.198.62.keytab.
Entry for principal kudu/127.10.198.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790376259650642-11032-0/krb5kdc/kudu_127.10.198.62.keytab.
I20260925 22:44:29.087353 11032 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrWXm8X/build/debug/bin/kudu
/tmp/dist-test-taskrWXm8X/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790376259650642-11032-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790376259650642-11032-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790376259650642-11032-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790376259650642-11032-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.198.62:42121
--webserver_interface=127.10.198.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790376259650642-11032-0/krb5kdc/kudu_127.10.198.62.keytab
--principal=kudu/127.10.198.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.198.20:44927
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.198.62:42121
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790376259650642-11032-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790376259650642-11032-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790376259650642-11032-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:44:29.210716 12731 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:44:29.211304 12731 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:44:29.211464 12731 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:44:29.216028 12731 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 22:44:29.216158 12731 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260925 22:44:29.216190 12731 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:44:29.216228 12731 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:44:29.216269 12731 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 22:44:29.216302 12731 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 22:44:29.222492 12731 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.198.20:44927
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790376259650642-11032-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790376259650642-11032-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.198.62:42121
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790376259650642-11032-0/krb5kdc/kudu_127.10.198.62.keytab
--principal=kudu/127.10.198.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.198.62:42121
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790376259650642-11032-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.198.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12731
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790376259650642-11032-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:09:16 UTC on ccycloud.quasar-afmhyw.root.comops.site
build id 4673379
I20260925 22:44:29.224061 12731 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:44:29.225299 12731 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:44:29.235297 12737 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:44:29.235445 12731 server_base.cc:1034] running on GCE node
W20260925 22:44:29.235224 12736 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:44:29.235226 12739 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:44:29.236303 12731 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:44:29.237648 12731 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:44:29.238893 12731 hybrid_clock.cc:648] HybridClock initialized: now 1790376269238866 us; error 39 us; skew 500 ppm
Sep 25 22:44:29 dist-test-slave-5vdr krb5kdc[12718](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376269, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.198.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:44:29.242313 12731 init.cc:377] Logged in from keytab as kudu/127.10.198.62@KRBTEST.COM (short username kudu)
I20260925 22:44:29.243531 12731 webserver.cc:460] Webserver started at http://127.10.198.62:39137/ using document root <none> and password file <none>
I20260925 22:44:29.244294 12731 fs_manager.cc:359] Metadata directory not provided
I20260925 22:44:29.244410 12731 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:44:29.244675 12731 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:44:29.246999 12731 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790376259650642-11032-0/minicluster-data/master-0/data/instance:
uuid: "9f0e816c575a42caac240acda127e507"
format_stamp: "Formatted at 2026-09-25 22:44:29 on dist-test-slave-5vdr"
server_key: "1e3bdf8d83f33eb537424c2ecdd26095"
server_key_iv: "1f27fa27fcb90d185524705cb45eeb90"
server_key_version: "encryptionkey@0"
I20260925 22:44:29.247687 12731 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790376259650642-11032-0/minicluster-data/master-0/wal/instance:
uuid: "9f0e816c575a42caac240acda127e507"
format_stamp: "Formatted at 2026-09-25 22:44:29 on dist-test-slave-5vdr"
server_key: "1e3bdf8d83f33eb537424c2ecdd26095"
server_key_iv: "1f27fa27fcb90d185524705cb45eeb90"
server_key_version: "encryptionkey@0"
I20260925 22:44:29.252421 12731 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.001s
I20260925 22:44:29.255501 12746 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:44:29.257169 12731 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 22:44:29.257401 12731 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790376259650642-11032-0/minicluster-data/master-0/data,/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790376259650642-11032-0/minicluster-data/master-0/wal
uuid: "9f0e816c575a42caac240acda127e507"
format_stamp: "Formatted at 2026-09-25 22:44:29 on dist-test-slave-5vdr"
server_key: "1e3bdf8d83f33eb537424c2ecdd26095"
server_key_iv: "1f27fa27fcb90d185524705cb45eeb90"
server_key_version: "encryptionkey@0"
I20260925 22:44:29.257570 12731 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790376259650642-11032-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790376259650642-11032-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790376259650642-11032-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:44:29.272261 12731 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:44:29.275933 12731 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:44:29.276191 12731 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:44:29.285637 12731 rpc_server.cc:307] RPC server started. Bound to: 127.10.198.62:42121
I20260925 22:44:29.285653 12808 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.198.62:42121 every 8 connection(s)
I20260925 22:44:29.286783 12731 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790376259650642-11032-0/minicluster-data/master-0/data/info.pb
I20260925 22:44:29.290066 12809 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:44:29.295424 11032 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrWXm8X/build/debug/bin/kudu as pid 12731
I20260925 22:44:29.295569 11032 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790376259650642-11032-0/minicluster-data/master-0/wal/instance
I20260925 22:44:29.295887 11032 external_mini_cluster.cc:1428] Setting key 3411f5a7a9d9149f1d686604e7f84abf
I20260925 22:44:29.297582 12809 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9f0e816c575a42caac240acda127e507: Bootstrap starting.
I20260925 22:44:29.300343 12809 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9f0e816c575a42caac240acda127e507: Neither blocks nor log segments found. Creating new log.
I20260925 22:44:29.301471 12809 log.cc:824] T 00000000000000000000000000000000 P 9f0e816c575a42caac240acda127e507: Log is configured to *not* fsync() on all Append() calls
I20260925 22:44:29.304476 12809 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9f0e816c575a42caac240acda127e507: No bootstrap required, opened a new log
Sep 25 22:44:29 dist-test-slave-5vdr krb5kdc[12718](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376269, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.198.62@KRBTEST.COM
I20260925 22:44:29.309715 12809 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9f0e816c575a42caac240acda127e507 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f0e816c575a42caac240acda127e507" member_type: VOTER last_known_addr { host: "127.10.198.62" port: 42121 } }
I20260925 22:44:29.310078 12809 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9f0e816c575a42caac240acda127e507 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:44:29.310182 12809 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9f0e816c575a42caac240acda127e507 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9f0e816c575a42caac240acda127e507, State: Initialized, Role: FOLLOWER
I20260925 22:44:29.310806 12809 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9f0e816c575a42caac240acda127e507 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f0e816c575a42caac240acda127e507" member_type: VOTER last_known_addr { host: "127.10.198.62" port: 42121 } }
I20260925 22:44:29.311031 12809 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9f0e816c575a42caac240acda127e507 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 22:44:29.311152 12809 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9f0e816c575a42caac240acda127e507 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 22:44:29.311281 12809 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9f0e816c575a42caac240acda127e507 [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:44:29.312770 12809 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9f0e816c575a42caac240acda127e507 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f0e816c575a42caac240acda127e507" member_type: VOTER last_known_addr { host: "127.10.198.62" port: 42121 } }
I20260925 22:44:29.313237 12809 leader_election.cc:302] T 00000000000000000000000000000000 P 9f0e816c575a42caac240acda127e507 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9f0e816c575a42caac240acda127e507; no voters:
I20260925 22:44:29.313784 12809 leader_election.cc:288] T 00000000000000000000000000000000 P 9f0e816c575a42caac240acda127e507 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 22:44:29.313963 12814 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9f0e816c575a42caac240acda127e507 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 22:44:29.314235 12814 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9f0e816c575a42caac240acda127e507 [term 1 LEADER]: Becoming Leader. State: Replica: 9f0e816c575a42caac240acda127e507, State: Running, Role: LEADER
I20260925 22:44:29.314636 12812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:44:29.297649 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42278 (local address 127.10.198.62:42121)
0925 22:44:29.298172 (+ 523us) server_negotiation.cc:206] Beginning negotiation
0925 22:44:29.298187 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0925 22:44:29.298225 (+ 38us) server_negotiation.cc:407] Connection header received
0925 22:44:29.298949 (+ 724us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:44:29.298968 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:44:29.299363 (+ 395us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:44:29.299775 (+ 412us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:44:29.300969 (+ 1194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:44:29.302230 (+ 1261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:44:29.303658 (+ 1428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:44:29.304298 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:44:29.304459 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:44:29.308160 (+ 3701us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:44:29.308167 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:44:29.308187 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:44:29.308229 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:44:29.310480 (+ 2251us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:44:29.311183 (+ 703us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:44:29.311191 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:44:29.311200 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:44:29.311390 (+ 190us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:44:29.311749 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:44:29.311754 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:44:29.311755 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:44:29.312442 (+ 687us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:44:29.312687 (+ 245us) server_negotiation.cc:1035] Waiting for connection context
0925 22:44:29.312981 (+ 294us) server_negotiation.cc:299] Negotiation successful
0925 22:44:29.313432 (+ 451us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":363,"thread_start_us":170,"threads_started":1}
I20260925 22:44:29.314752 12814 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9f0e816c575a42caac240acda127e507 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f0e816c575a42caac240acda127e507" member_type: VOTER last_known_addr { host: "127.10.198.62" port: 42121 } }
I20260925 22:44:29.315313 12809 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9f0e816c575a42caac240acda127e507 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 22:44:29.316952 12815 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9f0e816c575a42caac240acda127e507 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9f0e816c575a42caac240acda127e507" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f0e816c575a42caac240acda127e507" member_type: VOTER last_known_addr { host: "127.10.198.62" port: 42121 } } }
I20260925 22:44:29.317129 12815 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9f0e816c575a42caac240acda127e507 [sys.catalog]: This master's current role is: LEADER
I20260925 22:44:29.317554 12816 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9f0e816c575a42caac240acda127e507 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9f0e816c575a42caac240acda127e507. Latest consensus state: current_term: 1 leader_uuid: "9f0e816c575a42caac240acda127e507" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f0e816c575a42caac240acda127e507" member_type: VOTER last_known_addr { host: "127.10.198.62" port: 42121 } } }
I20260925 22:44:29.317663 12816 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9f0e816c575a42caac240acda127e507 [sys.catalog]: This master's current role is: LEADER
I20260925 22:44:29.318089 12823 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 22:44:29.322436 12823 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 22:44:29.329746 12823 catalog_manager.cc:1347] Generated new cluster ID: c5c51540f11540679960d476b717fdc8
I20260925 22:44:29.329846 12823 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 22:44:29.366235 12823 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 22:44:29.367348 12823 catalog_manager.cc:1504] Loading token signing keys...
I20260925 22:44:29.383898 12823 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9f0e816c575a42caac240acda127e507: Generated new TSK 0
I20260925 22:44:29.384938 12823 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.10.198.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.198.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.198.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790376259650642-11032-0/krb5kdc/kudu_127.10.198.1.keytab.
Entry for principal kudu/127.10.198.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790376259650642-11032-0/krb5kdc/kudu_127.10.198.1.keytab.
I20260925 22:44:29.428095 11032 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrWXm8X/build/debug/bin/kudu
/tmp/dist-test-taskrWXm8X/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790376259650642-11032-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790376259650642-11032-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790376259650642-11032-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790376259650642-11032-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.198.1:0
--local_ip_for_outbound_sockets=127.10.198.1
--webserver_interface=127.10.198.1
--webserver_port=0
--tserver_master_addrs=127.10.198.62:42121
--keytab_file=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790376259650642-11032-0/krb5kdc/kudu_127.10.198.1.keytab
--principal=kudu/127.10.198.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.198.20:44927
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790376259650642-11032-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790376259650642-11032-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790376259650642-11032-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:44:29.556870 12835 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:44:29.557240 12835 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:44:29.557324 12835 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:44:29.562054 12835 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:44:29.562167 12835 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:44:29.562258 12835 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260925 22:44:29.562305 12835 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260925 22:44:29.562381 12835 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.198.1
I20260925 22:44:29.568578 12835 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.198.20:44927
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790376259650642-11032-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790376259650642-11032-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790376259650642-11032-0/krb5kdc/kudu_127.10.198.1.keytab
--principal=kudu/127.10.198.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.198.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790376259650642-11032-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.198.1
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.10.198.62:42121
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12835
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.198.1
--log_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790376259650642-11032-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:09:16 UTC on ccycloud.quasar-afmhyw.root.comops.site
build id 4673379
I20260925 22:44:29.569880 12835 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:44:29.571102 12835 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:44:29.582037 12840 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:44:29.582162 12835 server_base.cc:1034] running on GCE node
W20260925 22:44:29.582053 12843 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:44:29.582037 12841 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:44:29.582862 12835 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:44:29.583666 12835 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:44:29.584897 12835 hybrid_clock.cc:648] HybridClock initialized: now 1790376269584848 us; error 58 us; skew 500 ppm
Sep 25 22:44:29 dist-test-slave-5vdr krb5kdc[12718](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376269, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.198.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:44:29.588184 12835 init.cc:377] Logged in from keytab as kudu/127.10.198.1@KRBTEST.COM (short username kudu)
I20260925 22:44:29.589263 12835 webserver.cc:460] Webserver started at http://127.10.198.1:42163/ using document root <none> and password file <none>
I20260925 22:44:29.589918 12835 fs_manager.cc:359] Metadata directory not provided
I20260925 22:44:29.589994 12835 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:44:29.590257 12835 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:44:29.592710 12835 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790376259650642-11032-0/minicluster-data/ts-0/data/instance:
uuid: "3cad46077ef143feae0d7ee5ac6852f4"
format_stamp: "Formatted at 2026-09-25 22:44:29 on dist-test-slave-5vdr"
server_key: "61e8413bc59bc61277ca3d83c72714e9"
server_key_iv: "2b9a8afec7e585ad9f3bcab47cd24f02"
server_key_version: "encryptionkey@0"
I20260925 22:44:29.593392 12835 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790376259650642-11032-0/minicluster-data/ts-0/wal/instance:
uuid: "3cad46077ef143feae0d7ee5ac6852f4"
format_stamp: "Formatted at 2026-09-25 22:44:29 on dist-test-slave-5vdr"
server_key: "61e8413bc59bc61277ca3d83c72714e9"
server_key_iv: "2b9a8afec7e585ad9f3bcab47cd24f02"
server_key_version: "encryptionkey@0"
I20260925 22:44:29.597942 12835 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260925 22:44:29.601361 12850 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:44:29.602977 12835 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260925 22:44:29.603178 12835 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790376259650642-11032-0/minicluster-data/ts-0/data,/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790376259650642-11032-0/minicluster-data/ts-0/wal
uuid: "3cad46077ef143feae0d7ee5ac6852f4"
format_stamp: "Formatted at 2026-09-25 22:44:29 on dist-test-slave-5vdr"
server_key: "61e8413bc59bc61277ca3d83c72714e9"
server_key_iv: "2b9a8afec7e585ad9f3bcab47cd24f02"
server_key_version: "encryptionkey@0"
I20260925 22:44:29.603345 12835 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790376259650642-11032-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790376259650642-11032-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790376259650642-11032-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:44:29.628330 12835 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:44:29.631834 12835 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:44:29.632134 12835 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:44:29.634042 12835 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:44:29.634135 12835 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:44:29.634199 12835 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:44:29.634253 12835 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 25 22:44:29 dist-test-slave-5vdr krb5kdc[12718](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376269, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.198.1@KRBTEST.COM for kudu/127.10.198.62@KRBTEST.COM
I20260925 22:44:29.645870 12835 rpc_server.cc:307] RPC server started. Bound to: 127.10.198.1:36901
I20260925 22:44:29.645905 12965 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.198.1:36901 every 8 connection(s)
I20260925 22:44:29.647240 12835 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790376259650642-11032-0/minicluster-data/ts-0/data/info.pb
I20260925 22:44:29.650054 12812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:44:29.634934 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.198.1:46253 (local address 127.10.198.62:42121)
0925 22:44:29.635102 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0925 22:44:29.635107 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:44:29.636217 (+ 1110us) server_negotiation.cc:407] Connection header received
0925 22:44:29.637493 (+ 1276us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:44:29.637500 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:44:29.637579 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:44:29.637698 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:44:29.639346 (+ 1648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:44:29.640300 (+ 954us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:44:29.641924 (+ 1624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:44:29.642182 (+ 258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:44:29.642316 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:44:29.645595 (+ 3279us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:44:29.645600 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:44:29.645603 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:44:29.645644 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:44:29.647472 (+ 1828us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:44:29.648411 (+ 939us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:44:29.648415 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:44:29.648417 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:44:29.648519 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:44:29.648922 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:44:29.648925 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:44:29.648927 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:44:29.649151 (+ 224us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:44:29.649243 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0925 22:44:29.649782 (+ 539us) server_negotiation.cc:299] Negotiation successful
0925 22:44:29.649913 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260925 22:44:29.651005 12859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:44:29.635469 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.198.62:42121 (local address 127.10.198.1:46253)
0925 22:44:29.636025 (+ 556us) negotiation.cc:107] Waiting for socket to connect
0925 22:44:29.636082 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0925 22:44:29.637243 (+ 1161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:44:29.637956 (+ 713us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:44:29.637964 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:44:29.638457 (+ 493us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:44:29.639162 (+ 705us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:44:29.639180 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:44:29.640477 (+ 1297us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:44:29.640482 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:44:29.641746 (+ 1264us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:44:29.641759 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:44:29.642379 (+ 620us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:44:29.642383 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:44:29.642622 (+ 239us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:44:29.643405 (+ 783us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:44:29.643438 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:44:29.645353 (+ 1915us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:44:29.647638 (+ 2285us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:44:29.647650 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:44:29.647669 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:44:29.648116 (+ 447us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:44:29.648621 (+ 505us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:44:29.648623 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:44:29.648624 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:44:29.648778 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:44:29.649255 (+ 477us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:44:29.649261 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:44:29.649538 (+ 277us) client_negotiation.cc:769] Sending connection context
0925 22:44:29.649704 (+ 166us) client_negotiation.cc:240] Negotiation successful
0925 22:44:29.650000 (+ 296us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":281,"thread_start_us":146,"threads_started":1}
I20260925 22:44:29.652729 12966 heartbeater.cc:347] Connected to a master server at 127.10.198.62:42121
I20260925 22:44:29.652971 12966 heartbeater.cc:464] Registering TS with master...
I20260925 22:44:29.653574 12966 heartbeater.cc:511] Master 127.10.198.62:42121 requested a full tablet report, sending...
I20260925 22:44:29.655362 12762 ts_manager.cc:194] Registered new tserver with Master: 3cad46077ef143feae0d7ee5ac6852f4 (127.10.198.1:36901)
I20260925 22:44:29.656636 12762 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.198.1@KRBTEST.COM'} at 127.10.198.1:46253
I20260925 22:44:29.656803 11032 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrWXm8X/build/debug/bin/kudu as pid 12835
I20260925 22:44:29.656915 11032 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790376259650642-11032-0/minicluster-data/ts-0/wal/instance
I20260925 22:44:29.657198 11032 external_mini_cluster.cc:1428] Setting key 4bc26b11efb1ec385de017a9ed0d3ec3
I20260925 22:44:29.664196 12972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:44:29.655131 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.198.62:42121 (local address 127.10.198.1:46869)
0925 22:44:29.655546 (+ 415us) negotiation.cc:107] Waiting for socket to connect
0925 22:44:29.655567 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0925 22:44:29.655688 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:44:29.656020 (+ 332us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:44:29.656023 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:44:29.656257 (+ 234us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:44:29.656362 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:44:29.656368 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:44:29.657450 (+ 1082us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:44:29.657454 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:44:29.658335 (+ 881us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:44:29.658347 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:44:29.658950 (+ 603us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:44:29.658960 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:44:29.659052 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:44:29.659699 (+ 647us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:44:29.659712 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:44:29.660184 (+ 472us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:44:29.662420 (+ 2236us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:44:29.662424 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:44:29.662425 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:44:29.662651 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:44:29.663035 (+ 384us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:44:29.663037 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:44:29.663039 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:44:29.663129 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:44:29.663645 (+ 516us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:44:29.663647 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:44:29.663740 (+ 93us) client_negotiation.cc:769] Sending connection context
0925 22:44:29.663829 (+ 89us) client_negotiation.cc:240] Negotiation successful
0925 22:44:29.664028 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":300,"thread_start_us":108,"threads_started":1}
I20260925 22:44:29.664378 12812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:44:29.655202 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.198.1:46869 (local address 127.10.198.62:42121)
0925 22:44:29.655353 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0925 22:44:29.655358 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:44:29.655641 (+ 283us) server_negotiation.cc:407] Connection header received
0925 22:44:29.655799 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:44:29.655804 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:44:29.655898 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:44:29.656003 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:44:29.656492 (+ 489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:44:29.657284 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:44:29.658479 (+ 1195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:44:29.658789 (+ 310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:44:29.658922 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:44:29.660365 (+ 1443us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:44:29.660372 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:44:29.660378 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:44:29.660414 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:44:29.662281 (+ 1867us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:44:29.662802 (+ 521us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:44:29.662808 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:44:29.662813 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:44:29.662929 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:44:29.663236 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:44:29.663243 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:44:29.663247 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:44:29.663534 (+ 287us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:44:29.663659 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0925 22:44:29.663987 (+ 328us) server_negotiation.cc:299] Negotiation successful
0925 22:44:29.664136 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":5,"server-negotiator.queue_time_us":62}
WARNING: no policy specified for kudu/127.10.198.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.198.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.198.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790376259650642-11032-0/krb5kdc/kudu_127.10.198.2.keytab.
Entry for principal kudu/127.10.198.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790376259650642-11032-0/krb5kdc/kudu_127.10.198.2.keytab.
I20260925 22:44:29.688119 11032 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrWXm8X/build/debug/bin/kudu
/tmp/dist-test-taskrWXm8X/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790376259650642-11032-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790376259650642-11032-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790376259650642-11032-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790376259650642-11032-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.198.2:0
--local_ip_for_outbound_sockets=127.10.198.2
--webserver_interface=127.10.198.2
--webserver_port=0
--tserver_master_addrs=127.10.198.62:42121
--keytab_file=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790376259650642-11032-0/krb5kdc/kudu_127.10.198.2.keytab
--principal=kudu/127.10.198.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.198.20:44927
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790376259650642-11032-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790376259650642-11032-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790376259650642-11032-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:44:29.812354 12975 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:44:29.812664 12975 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:44:29.812714 12975 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:44:29.817797 12975 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:44:29.817880 12975 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:44:29.817925 12975 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260925 22:44:29.817970 12975 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260925 22:44:29.818037 12975 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.198.2
I20260925 22:44:29.823935 12975 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.198.20:44927
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790376259650642-11032-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790376259650642-11032-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790376259650642-11032-0/krb5kdc/kudu_127.10.198.2.keytab
--principal=kudu/127.10.198.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.198.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790376259650642-11032-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.198.2
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.10.198.62:42121
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12975
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.198.2
--log_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790376259650642-11032-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:09:16 UTC on ccycloud.quasar-afmhyw.root.comops.site
build id 4673379
I20260925 22:44:29.825141 12975 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:44:29.826277 12975 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:44:29.837186 12983 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:44:29.837270 12975 server_base.cc:1034] running on GCE node
W20260925 22:44:29.837190 12980 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:44:29.837186 12981 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:44:29.837927 12975 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:44:29.838685 12975 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:44:29.839941 12975 hybrid_clock.cc:648] HybridClock initialized: now 1790376269839923 us; error 35 us; skew 500 ppm
Sep 25 22:44:29 dist-test-slave-5vdr krb5kdc[12718](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376269, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.198.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:44:29.842981 12975 init.cc:377] Logged in from keytab as kudu/127.10.198.2@KRBTEST.COM (short username kudu)
I20260925 22:44:29.844164 12975 webserver.cc:460] Webserver started at http://127.10.198.2:43355/ using document root <none> and password file <none>
I20260925 22:44:29.844810 12975 fs_manager.cc:359] Metadata directory not provided
I20260925 22:44:29.844883 12975 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:44:29.845067 12975 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:44:29.847316 12975 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790376259650642-11032-0/minicluster-data/ts-1/data/instance:
uuid: "2a585751d44c49f98359ac6423e39c71"
format_stamp: "Formatted at 2026-09-25 22:44:29 on dist-test-slave-5vdr"
server_key: "7605b5a56fb47e75f29e70ffb3ebbdbb"
server_key_iv: "fa2e1cd960f0eb1b5ec00839de230f7b"
server_key_version: "encryptionkey@0"
I20260925 22:44:29.848003 12975 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790376259650642-11032-0/minicluster-data/ts-1/wal/instance:
uuid: "2a585751d44c49f98359ac6423e39c71"
format_stamp: "Formatted at 2026-09-25 22:44:29 on dist-test-slave-5vdr"
server_key: "7605b5a56fb47e75f29e70ffb3ebbdbb"
server_key_iv: "fa2e1cd960f0eb1b5ec00839de230f7b"
server_key_version: "encryptionkey@0"
I20260925 22:44:29.852459 12975 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260925 22:44:29.856009 12990 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:44:29.857514 12975 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260925 22:44:29.857704 12975 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790376259650642-11032-0/minicluster-data/ts-1/data,/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790376259650642-11032-0/minicluster-data/ts-1/wal
uuid: "2a585751d44c49f98359ac6423e39c71"
format_stamp: "Formatted at 2026-09-25 22:44:29 on dist-test-slave-5vdr"
server_key: "7605b5a56fb47e75f29e70ffb3ebbdbb"
server_key_iv: "fa2e1cd960f0eb1b5ec00839de230f7b"
server_key_version: "encryptionkey@0"
I20260925 22:44:29.857867 12975 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790376259650642-11032-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790376259650642-11032-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790376259650642-11032-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:44:29.869177 12975 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:44:29.872938 12975 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:44:29.873170 12975 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:44:29.874758 12975 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:44:29.874859 12975 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:44:29.874935 12975 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:44:29.875020 12975 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 25 22:44:29 dist-test-slave-5vdr krb5kdc[12718](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376269, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.198.2@KRBTEST.COM for kudu/127.10.198.62@KRBTEST.COM
I20260925 22:44:29.887714 12975 rpc_server.cc:307] RPC server started. Bound to: 127.10.198.2:45425
I20260925 22:44:29.887785 13105 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.198.2:45425 every 8 connection(s)
I20260925 22:44:29.888960 12975 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790376259650642-11032-0/minicluster-data/ts-1/data/info.pb
I20260925 22:44:29.891129 12812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:44:29.875981 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.198.2:43887 (local address 127.10.198.62:42121)
0925 22:44:29.876118 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0925 22:44:29.876121 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 22:44:29.877414 (+ 1293us) server_negotiation.cc:407] Connection header received
0925 22:44:29.878774 (+ 1360us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:44:29.878778 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:44:29.878848 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:44:29.878981 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:44:29.880661 (+ 1680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:44:29.881291 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:44:29.882756 (+ 1465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:44:29.883060 (+ 304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:44:29.883204 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:44:29.886949 (+ 3745us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:44:29.886953 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:44:29.886955 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:44:29.886986 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:44:29.888771 (+ 1785us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:44:29.889479 (+ 708us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:44:29.889483 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:44:29.889485 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:44:29.889588 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:44:29.890051 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:44:29.890054 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:44:29.890056 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:44:29.890296 (+ 240us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:44:29.890383 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0925 22:44:29.890881 (+ 498us) server_negotiation.cc:299] Negotiation successful
0925 22:44:29.890998 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260925 22:44:29.892174 12999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:44:29.876650 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.198.62:42121 (local address 127.10.198.2:43887)
0925 22:44:29.877226 (+ 576us) negotiation.cc:107] Waiting for socket to connect
0925 22:44:29.877276 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0925 22:44:29.878487 (+ 1211us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:44:29.879219 (+ 732us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:44:29.879229 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:44:29.879884 (+ 655us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:44:29.880479 (+ 595us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:44:29.880496 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:44:29.881438 (+ 942us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:44:29.881442 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:44:29.882608 (+ 1166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:44:29.882621 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:44:29.883272 (+ 651us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:44:29.883277 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:44:29.883524 (+ 247us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:44:29.884326 (+ 802us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:44:29.884357 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:44:29.886668 (+ 2311us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:44:29.888941 (+ 2273us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:44:29.888947 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:44:29.888964 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:44:29.889333 (+ 369us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:44:29.889706 (+ 373us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:44:29.889709 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:44:29.889711 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:44:29.889886 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:44:29.890400 (+ 514us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:44:29.890405 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:44:29.890655 (+ 250us) client_negotiation.cc:769] Sending connection context
0925 22:44:29.890812 (+ 157us) client_negotiation.cc:240] Negotiation successful
0925 22:44:29.891106 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":363,"thread_start_us":173,"threads_started":1}
I20260925 22:44:29.893591 13106 heartbeater.cc:347] Connected to a master server at 127.10.198.62:42121
I20260925 22:44:29.893774 13106 heartbeater.cc:464] Registering TS with master...
I20260925 22:44:29.894366 13106 heartbeater.cc:511] Master 127.10.198.62:42121 requested a full tablet report, sending...
I20260925 22:44:29.895579 12763 ts_manager.cc:194] Registered new tserver with Master: 2a585751d44c49f98359ac6423e39c71 (127.10.198.2:45425)
I20260925 22:44:29.895638 11032 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrWXm8X/build/debug/bin/kudu as pid 12975
I20260925 22:44:29.895748 11032 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790376259650642-11032-0/minicluster-data/ts-1/wal/instance
I20260925 22:44:29.896107 11032 external_mini_cluster.cc:1428] Setting key 5c2f9f8f459e545fd8b45ad599c19791
I20260925 22:44:29.896430 12763 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.198.2@KRBTEST.COM'} at 127.10.198.2:43887
I20260925 22:44:29.904420 12812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:44:29.896071 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.198.2:32903 (local address 127.10.198.62:42121)
0925 22:44:29.896185 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0925 22:44:29.896190 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:44:29.896486 (+ 296us) server_negotiation.cc:407] Connection header received
0925 22:44:29.896658 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:44:29.896661 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:44:29.896708 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:44:29.896827 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:44:29.897395 (+ 568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:44:29.898131 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:44:29.899467 (+ 1336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:44:29.899694 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:44:29.899814 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:44:29.901063 (+ 1249us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:44:29.901066 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:44:29.901068 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:44:29.901093 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:44:29.902487 (+ 1394us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:44:29.903000 (+ 513us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:44:29.903007 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:44:29.903009 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:44:29.903101 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:44:29.903412 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:44:29.903416 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:44:29.903418 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:44:29.903669 (+ 251us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:44:29.903789 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0925 22:44:29.904116 (+ 327us) server_negotiation.cc:299] Negotiation successful
0925 22:44:29.904238 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260925 22:44:29.904435 13112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:44:29.896002 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.198.62:42121 (local address 127.10.198.2:32903)
0925 22:44:29.896381 (+ 379us) negotiation.cc:107] Waiting for socket to connect
0925 22:44:29.896398 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 22:44:29.896507 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:44:29.896877 (+ 370us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:44:29.896879 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:44:29.897128 (+ 249us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:44:29.897265 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:44:29.897274 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:44:29.898293 (+ 1019us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:44:29.898299 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:44:29.899314 (+ 1015us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:44:29.899325 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:44:29.899817 (+ 492us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:44:29.899821 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:44:29.899949 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:44:29.900537 (+ 588us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:44:29.900549 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:44:29.900898 (+ 349us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:44:29.902641 (+ 1743us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:44:29.902644 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:44:29.902646 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:44:29.902883 (+ 237us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:44:29.903210 (+ 327us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:44:29.903213 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:44:29.903215 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:44:29.903308 (+ 93us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:44:29.903787 (+ 479us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:44:29.903790 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:44:29.903931 (+ 141us) client_negotiation.cc:769] Sending connection context
0925 22:44:29.904053 (+ 122us) client_negotiation.cc:240] Negotiation successful
0925 22:44:29.904230 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":293,"thread_start_us":117,"threads_started":1}
WARNING: no policy specified for kudu/127.10.198.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.198.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.198.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790376259650642-11032-0/krb5kdc/kudu_127.10.198.3.keytab.
Entry for principal kudu/127.10.198.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790376259650642-11032-0/krb5kdc/kudu_127.10.198.3.keytab.
I20260925 22:44:29.927112 11032 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrWXm8X/build/debug/bin/kudu
/tmp/dist-test-taskrWXm8X/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790376259650642-11032-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790376259650642-11032-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790376259650642-11032-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790376259650642-11032-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.198.3:0
--local_ip_for_outbound_sockets=127.10.198.3
--webserver_interface=127.10.198.3
--webserver_port=0
--tserver_master_addrs=127.10.198.62:42121
--keytab_file=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790376259650642-11032-0/krb5kdc/kudu_127.10.198.3.keytab
--principal=kudu/127.10.198.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.198.20:44927
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790376259650642-11032-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790376259650642-11032-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790376259650642-11032-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:44:30.048834 13115 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:44:30.049213 13115 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:44:30.049301 13115 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:44:30.055047 13115 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:44:30.055173 13115 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:44:30.055258 13115 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260925 22:44:30.055305 13115 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260925 22:44:30.055367 13115 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.198.3
I20260925 22:44:30.061767 13115 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.198.20:44927
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790376259650642-11032-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790376259650642-11032-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790376259650642-11032-0/krb5kdc/kudu_127.10.198.3.keytab
--principal=kudu/127.10.198.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.198.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790376259650642-11032-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.198.3
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.10.198.62:42121
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13115
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.198.3
--log_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790376259650642-11032-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:09:16 UTC on ccycloud.quasar-afmhyw.root.comops.site
build id 4673379
I20260925 22:44:30.063139 13115 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:44:30.064350 13115 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:44:30.075330 13123 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:44:30.075320 13121 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:44:30.075320 13120 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:44:30.075657 13115 server_base.cc:1034] running on GCE node
I20260925 22:44:30.076188 13115 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:44:30.077116 13115 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:44:30.080506 13115 hybrid_clock.cc:648] HybridClock initialized: now 1790376270080461 us; error 45 us; skew 500 ppm
Sep 25 22:44:30 dist-test-slave-5vdr krb5kdc[12718](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376270, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.198.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:44:30.084117 13115 init.cc:377] Logged in from keytab as kudu/127.10.198.3@KRBTEST.COM (short username kudu)
I20260925 22:44:30.085186 13115 webserver.cc:460] Webserver started at http://127.10.198.3:33123/ using document root <none> and password file <none>
I20260925 22:44:30.085800 13115 fs_manager.cc:359] Metadata directory not provided
I20260925 22:44:30.085871 13115 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:44:30.086061 13115 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:44:30.088490 13115 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790376259650642-11032-0/minicluster-data/ts-2/data/instance:
uuid: "c1b2193f640945a895e6497c6422849f"
format_stamp: "Formatted at 2026-09-25 22:44:30 on dist-test-slave-5vdr"
server_key: "c1399f6f723e273edd52aaddd8caddcc"
server_key_iv: "5745779a72f70f404034d1484188ba81"
server_key_version: "encryptionkey@0"
I20260925 22:44:30.089116 13115 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790376259650642-11032-0/minicluster-data/ts-2/wal/instance:
uuid: "c1b2193f640945a895e6497c6422849f"
format_stamp: "Formatted at 2026-09-25 22:44:30 on dist-test-slave-5vdr"
server_key: "c1399f6f723e273edd52aaddd8caddcc"
server_key_iv: "5745779a72f70f404034d1484188ba81"
server_key_version: "encryptionkey@0"
I20260925 22:44:30.093683 13115 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260925 22:44:30.096873 13130 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:44:30.098426 13115 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 22:44:30.098567 13115 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790376259650642-11032-0/minicluster-data/ts-2/data,/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790376259650642-11032-0/minicluster-data/ts-2/wal
uuid: "c1b2193f640945a895e6497c6422849f"
format_stamp: "Formatted at 2026-09-25 22:44:30 on dist-test-slave-5vdr"
server_key: "c1399f6f723e273edd52aaddd8caddcc"
server_key_iv: "5745779a72f70f404034d1484188ba81"
server_key_version: "encryptionkey@0"
I20260925 22:44:30.098686 13115 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790376259650642-11032-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790376259650642-11032-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790376259650642-11032-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:44:30.110064 13115 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:44:30.113760 13115 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:44:30.113945 13115 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:44:30.115648 13115 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:44:30.115713 13115 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:44:30.115759 13115 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:44:30.115954 13115 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
I20260925 22:44:30.127480 13115 rpc_server.cc:307] RPC server started. Bound to: 127.10.198.3:41889
I20260925 22:44:30.127568 13245 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.198.3:41889 every 8 connection(s)
Sep 25 22:44:30 dist-test-slave-5vdr krb5kdc[12718](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376270, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.198.3@KRBTEST.COM for kudu/127.10.198.62@KRBTEST.COM
I20260925 22:44:30.128827 13115 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790376259650642-11032-0/minicluster-data/ts-2/data/info.pb
I20260925 22:44:30.133533 12812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:44:30.116516 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.198.3:42575 (local address 127.10.198.62:42121)
0925 22:44:30.116679 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0925 22:44:30.116684 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:44:30.118423 (+ 1739us) server_negotiation.cc:407] Connection header received
0925 22:44:30.120074 (+ 1651us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:44:30.120081 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:44:30.120162 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:44:30.120283 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:44:30.122301 (+ 2018us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:44:30.123261 (+ 960us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:44:30.125483 (+ 2222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:44:30.125809 (+ 326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:44:30.125976 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:44:30.129532 (+ 3556us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:44:30.129537 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:44:30.129540 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:44:30.129575 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:44:30.131070 (+ 1495us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:44:30.131729 (+ 659us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:44:30.131733 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:44:30.131734 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:44:30.131864 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:44:30.132277 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:44:30.132280 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:44:30.132281 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:44:30.132522 (+ 241us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:44:30.132609 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0925 22:44:30.133245 (+ 636us) server_negotiation.cc:299] Negotiation successful
0925 22:44:30.133378 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260925 22:44:30.134474 13139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:44:30.117504 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.198.62:42121 (local address 127.10.198.3:42575)
0925 22:44:30.118214 (+ 710us) negotiation.cc:107] Waiting for socket to connect
0925 22:44:30.118291 (+ 77us) client_negotiation.cc:174] Beginning negotiation
0925 22:44:30.119729 (+ 1438us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:44:30.120545 (+ 816us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:44:30.120561 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:44:30.121183 (+ 622us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:44:30.122056 (+ 873us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:44:30.122084 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:44:30.123470 (+ 1386us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:44:30.123474 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:44:30.125260 (+ 1786us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:44:30.125289 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:44:30.126027 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:44:30.126032 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:44:30.126293 (+ 261us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:44:30.127054 (+ 761us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:44:30.127074 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:44:30.129233 (+ 2159us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:44:30.131218 (+ 1985us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:44:30.131224 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:44:30.131240 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:44:30.131566 (+ 326us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:44:30.131975 (+ 409us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:44:30.131978 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:44:30.131980 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:44:30.132158 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:44:30.132646 (+ 488us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:44:30.132653 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:44:30.132963 (+ 310us) client_negotiation.cc:769] Sending connection context
0925 22:44:30.133139 (+ 176us) client_negotiation.cc:240] Negotiation successful
0925 22:44:30.133469 (+ 330us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":375,"thread_start_us":208,"threads_started":1}
I20260925 22:44:30.135092 11032 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrWXm8X/build/debug/bin/kudu as pid 13115
I20260925 22:44:30.135259 11032 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790376259650642-11032-0/minicluster-data/ts-2/wal/instance
I20260925 22:44:30.135614 11032 external_mini_cluster.cc:1428] Setting key eb13b54558140d14f77880f7f2e0f7e6
I20260925 22:44:30.136121 13246 heartbeater.cc:347] Connected to a master server at 127.10.198.62:42121
I20260925 22:44:30.136284 13246 heartbeater.cc:464] Registering TS with master...
I20260925 22:44:30.137012 13246 heartbeater.cc:511] Master 127.10.198.62:42121 requested a full tablet report, sending...
I20260925 22:44:30.138124 12762 ts_manager.cc:194] Registered new tserver with Master: c1b2193f640945a895e6497c6422849f (127.10.198.3:41889)
I20260925 22:44:30.138945 12762 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.198.3@KRBTEST.COM'} at 127.10.198.3:42575
I20260925 22:44:30.146184 12812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:44:30.138665 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.198.3:44457 (local address 127.10.198.62:42121)
0925 22:44:30.138797 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0925 22:44:30.138802 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:44:30.138952 (+ 150us) server_negotiation.cc:407] Connection header received
0925 22:44:30.139062 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:44:30.139065 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:44:30.139105 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:44:30.139194 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:44:30.139734 (+ 540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:44:30.140373 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:44:30.141236 (+ 863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:44:30.141472 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:44:30.141590 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:44:30.142822 (+ 1232us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:44:30.142826 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:44:30.142829 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:44:30.142865 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:44:30.144405 (+ 1540us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:44:30.144916 (+ 511us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:44:30.144920 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:44:30.144921 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:44:30.145003 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:44:30.145295 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:44:30.145299 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:44:30.145301 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:44:30.145503 (+ 202us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:44:30.145615 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0925 22:44:30.145885 (+ 270us) server_negotiation.cc:299] Negotiation successful
0925 22:44:30.146008 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260925 22:44:30.146198 13252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:44:30.138546 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.198.62:42121 (local address 127.10.198.3:44457)
0925 22:44:30.138860 (+ 314us) negotiation.cc:107] Waiting for socket to connect
0925 22:44:30.138874 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 22:44:30.138971 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:44:30.139252 (+ 281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:44:30.139257 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:44:30.139496 (+ 239us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:44:30.139609 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:44:30.139616 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:44:30.140515 (+ 899us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:44:30.140518 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:44:30.141102 (+ 584us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:44:30.141111 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:44:30.141594 (+ 483us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:44:30.141597 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:44:30.141700 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:44:30.142298 (+ 598us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:44:30.142312 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:44:30.142657 (+ 345us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:44:30.144562 (+ 1905us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:44:30.144566 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:44:30.144569 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:44:30.144789 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:44:30.145102 (+ 313us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:44:30.145105 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:44:30.145107 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:44:30.145198 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:44:30.145603 (+ 405us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:44:30.145606 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:44:30.145715 (+ 109us) client_negotiation.cc:769] Sending connection context
0925 22:44:30.145826 (+ 111us) client_negotiation.cc:240] Negotiation successful
0925 22:44:30.146011 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":224,"thread_start_us":102,"threads_started":1}
I20260925 22:44:30.149835 11032 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 25 22:44:30 dist-test-slave-5vdr krb5kdc[12718](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376270, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 25 22:44:30 dist-test-slave-5vdr krb5kdc[12718](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376270, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.198.62@KRBTEST.COM
I20260925 22:44:30.177968 12812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:44:30.168700 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42286 (local address 127.10.198.62:42121)
0925 22:44:30.168835 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0925 22:44:30.168839 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:44:30.168851 (+ 12us) server_negotiation.cc:407] Connection header received
0925 22:44:30.168895 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:44:30.168897 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:44:30.168938 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:44:30.169039 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:44:30.169829 (+ 790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:44:30.170368 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:44:30.171338 (+ 970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:44:30.171567 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:44:30.171689 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:44:30.174444 (+ 2755us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:44:30.174448 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:44:30.174450 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:44:30.174488 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:44:30.176113 (+ 1625us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:44:30.176637 (+ 524us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:44:30.176641 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:44:30.176643 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:44:30.176743 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:44:30.177050 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:44:30.177054 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:44:30.177056 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:44:30.177265 (+ 209us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:44:30.177412 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0925 22:44:30.177668 (+ 256us) server_negotiation.cc:299] Negotiation successful
0925 22:44:30.177795 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260925 22:44:30.181257 12762 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42286:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 22:44:30.184502 12762 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 22:44:30.200515 13263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:44:30.194592 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.198.2:45425 (local address 127.0.0.1:33700)
0925 22:44:30.194972 (+ 380us) negotiation.cc:107] Waiting for socket to connect
0925 22:44:30.195028 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0925 22:44:30.195173 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:44:30.195970 (+ 797us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:44:30.195976 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:44:30.196004 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:44:30.196201 (+ 197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:44:30.196210 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:44:30.197656 (+ 1446us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:44:30.197659 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:44:30.198665 (+ 1006us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:44:30.198677 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:44:30.199886 (+ 1209us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:44:30.199890 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:44:30.200023 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:44:30.200069 (+ 46us) client_negotiation.cc:769] Sending connection context
0925 22:44:30.200191 (+ 122us) client_negotiation.cc:240] Negotiation successful
0925 22:44:30.200341 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":100,"threads_started":1}
I20260925 22:44:30.201136 13264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:44:30.194720 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33700 (local address 127.10.198.2:45425)
0925 22:44:30.195228 (+ 508us) server_negotiation.cc:206] Beginning negotiation
0925 22:44:30.195242 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0925 22:44:30.195293 (+ 51us) server_negotiation.cc:407] Connection header received
0925 22:44:30.195375 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:44:30.195381 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:44:30.195648 (+ 267us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:44:30.196002 (+ 354us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:44:30.196364 (+ 362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:44:30.197534 (+ 1170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:44:30.198831 (+ 1297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:44:30.199704 (+ 873us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:44:30.199875 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:44:30.200069 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0925 22:44:30.200875 (+ 806us) server_negotiation.cc:299] Negotiation successful
0925 22:44:30.200987 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":404,"thread_start_us":143,"threads_started":1}
I20260925 22:44:30.202023 13265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:44:30.196036 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.198.3:41889 (local address 127.0.0.1:45840)
0925 22:44:30.196325 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0925 22:44:30.196342 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 22:44:30.196435 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:44:30.197465 (+ 1030us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:44:30.197468 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:44:30.197495 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:44:30.197607 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:44:30.197614 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:44:30.199207 (+ 1593us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:44:30.199243 (+ 36us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:44:30.200404 (+ 1161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:44:30.200414 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:44:30.201613 (+ 1199us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:44:30.201616 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:44:30.201709 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:44:30.201732 (+ 23us) client_negotiation.cc:769] Sending connection context
0925 22:44:30.201802 (+ 70us) client_negotiation.cc:240] Negotiation successful
0925 22:44:30.201881 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":217,"mutex_wait_us":49,"thread_start_us":148,"threads_started":1}
I20260925 22:44:30.203209 13266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:44:30.196210 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.198.1:36901 (local address 127.0.0.1:38708)
0925 22:44:30.196743 (+ 533us) negotiation.cc:107] Waiting for socket to connect
0925 22:44:30.196757 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 22:44:30.196851 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:44:30.197644 (+ 793us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:44:30.197647 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:44:30.197664 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:44:30.197737 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:44:30.197743 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:44:30.199207 (+ 1464us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:44:30.199211 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:44:30.201117 (+ 1906us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:44:30.201129 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:44:30.202638 (+ 1509us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:44:30.202641 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:44:30.202756 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:44:30.202792 (+ 36us) client_negotiation.cc:769] Sending connection context
0925 22:44:30.202872 (+ 80us) client_negotiation.cc:240] Negotiation successful
0925 22:44:30.202958 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":472,"mutex_wait_us":44,"thread_start_us":112,"threads_started":1}
I20260925 22:44:30.203218 13268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:44:30.196636 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38708 (local address 127.10.198.1:36901)
0925 22:44:30.197081 (+ 445us) server_negotiation.cc:206] Beginning negotiation
0925 22:44:30.197088 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 22:44:30.197109 (+ 21us) server_negotiation.cc:407] Connection header received
0925 22:44:30.197179 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:44:30.197184 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:44:30.197371 (+ 187us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:44:30.197501 (+ 130us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:44:30.197879 (+ 378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:44:30.199060 (+ 1181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:44:30.201383 (+ 2323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:44:30.202256 (+ 873us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:44:30.202403 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:44:30.202559 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0925 22:44:30.202968 (+ 409us) server_negotiation.cc:299] Negotiation successful
0925 22:44:30.203085 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":349,"thread_start_us":94,"threads_started":1}
I20260925 22:44:30.203567 13267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:44:30.196560 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45840 (local address 127.10.198.3:41889)
0925 22:44:30.197041 (+ 481us) server_negotiation.cc:206] Beginning negotiation
0925 22:44:30.197051 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 22:44:30.197074 (+ 23us) server_negotiation.cc:407] Connection header received
0925 22:44:30.197140 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:44:30.197145 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:44:30.197317 (+ 172us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:44:30.197498 (+ 181us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:44:30.197735 (+ 237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:44:30.199060 (+ 1325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:44:30.200553 (+ 1493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:44:30.201464 (+ 911us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:44:30.201598 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:44:30.201776 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0925 22:44:30.201847 (+ 71us) server_negotiation.cc:299] Negotiation successful
0925 22:44:30.201944 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":368,"thread_start_us":128,"threads_started":1}
I20260925 22:44:30.204726 13040 tablet_service.cc:1467] Processing CreateTablet for tablet bf53324bfb8f4602b82b33b495239d3f (DEFAULT_TABLE table=test-table [id=c5a50e0d9a7e47cd9e897abd2ea9787f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 22:44:30.205812 13040 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bf53324bfb8f4602b82b33b495239d3f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:44:30.206609 13180 tablet_service.cc:1467] Processing CreateTablet for tablet bf53324bfb8f4602b82b33b495239d3f (DEFAULT_TABLE table=test-table [id=c5a50e0d9a7e47cd9e897abd2ea9787f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 22:44:30.206606 12899 tablet_service.cc:1467] Processing CreateTablet for tablet bf53324bfb8f4602b82b33b495239d3f (DEFAULT_TABLE table=test-table [id=c5a50e0d9a7e47cd9e897abd2ea9787f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 22:44:30.207787 12899 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bf53324bfb8f4602b82b33b495239d3f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:44:30.207785 13180 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bf53324bfb8f4602b82b33b495239d3f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:44:30.213665 13269 tablet_bootstrap.cc:492] T bf53324bfb8f4602b82b33b495239d3f P 2a585751d44c49f98359ac6423e39c71: Bootstrap starting.
I20260925 22:44:30.216233 13269 tablet_bootstrap.cc:654] T bf53324bfb8f4602b82b33b495239d3f P 2a585751d44c49f98359ac6423e39c71: Neither blocks nor log segments found. Creating new log.
I20260925 22:44:30.216687 13271 tablet_bootstrap.cc:492] T bf53324bfb8f4602b82b33b495239d3f P c1b2193f640945a895e6497c6422849f: Bootstrap starting.
I20260925 22:44:30.217340 13270 tablet_bootstrap.cc:492] T bf53324bfb8f4602b82b33b495239d3f P 3cad46077ef143feae0d7ee5ac6852f4: Bootstrap starting.
I20260925 22:44:30.217447 13269 log.cc:824] T bf53324bfb8f4602b82b33b495239d3f P 2a585751d44c49f98359ac6423e39c71: Log is configured to *not* fsync() on all Append() calls
I20260925 22:44:30.219380 13271 tablet_bootstrap.cc:654] T bf53324bfb8f4602b82b33b495239d3f P c1b2193f640945a895e6497c6422849f: Neither blocks nor log segments found. Creating new log.
I20260925 22:44:30.219697 13270 tablet_bootstrap.cc:654] T bf53324bfb8f4602b82b33b495239d3f P 3cad46077ef143feae0d7ee5ac6852f4: Neither blocks nor log segments found. Creating new log.
I20260925 22:44:30.220381 13271 log.cc:824] T bf53324bfb8f4602b82b33b495239d3f P c1b2193f640945a895e6497c6422849f: Log is configured to *not* fsync() on all Append() calls
I20260925 22:44:30.220439 13270 log.cc:824] T bf53324bfb8f4602b82b33b495239d3f P 3cad46077ef143feae0d7ee5ac6852f4: Log is configured to *not* fsync() on all Append() calls
I20260925 22:44:30.220551 13269 tablet_bootstrap.cc:492] T bf53324bfb8f4602b82b33b495239d3f P 2a585751d44c49f98359ac6423e39c71: No bootstrap required, opened a new log
I20260925 22:44:30.220773 13269 ts_tablet_manager.cc:1397] T bf53324bfb8f4602b82b33b495239d3f P 2a585751d44c49f98359ac6423e39c71: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260925 22:44:30.222777 13270 tablet_bootstrap.cc:492] T bf53324bfb8f4602b82b33b495239d3f P 3cad46077ef143feae0d7ee5ac6852f4: No bootstrap required, opened a new log
I20260925 22:44:30.223011 13270 ts_tablet_manager.cc:1397] T bf53324bfb8f4602b82b33b495239d3f P 3cad46077ef143feae0d7ee5ac6852f4: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20260925 22:44:30.224321 13271 tablet_bootstrap.cc:492] T bf53324bfb8f4602b82b33b495239d3f P c1b2193f640945a895e6497c6422849f: No bootstrap required, opened a new log
I20260925 22:44:30.224565 13271 ts_tablet_manager.cc:1397] T bf53324bfb8f4602b82b33b495239d3f P c1b2193f640945a895e6497c6422849f: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.000s
I20260925 22:44:30.225018 13269 raft_consensus.cc:348] T bf53324bfb8f4602b82b33b495239d3f P 2a585751d44c49f98359ac6423e39c71 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a585751d44c49f98359ac6423e39c71" member_type: VOTER last_known_addr { host: "127.10.198.2" port: 45425 } } peers { permanent_uuid: "c1b2193f640945a895e6497c6422849f" member_type: VOTER last_known_addr { host: "127.10.198.3" port: 41889 } } peers { permanent_uuid: "3cad46077ef143feae0d7ee5ac6852f4" member_type: VOTER last_known_addr { host: "127.10.198.1" port: 36901 } }
I20260925 22:44:30.225598 13269 raft_consensus.cc:374] T bf53324bfb8f4602b82b33b495239d3f P 2a585751d44c49f98359ac6423e39c71 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:44:30.225674 13269 raft_consensus.cc:729] T bf53324bfb8f4602b82b33b495239d3f P 2a585751d44c49f98359ac6423e39c71 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2a585751d44c49f98359ac6423e39c71, State: Initialized, Role: FOLLOWER
I20260925 22:44:30.226226 13269 consensus_queue.cc:260] T bf53324bfb8f4602b82b33b495239d3f P 2a585751d44c49f98359ac6423e39c71 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a585751d44c49f98359ac6423e39c71" member_type: VOTER last_known_addr { host: "127.10.198.2" port: 45425 } } peers { permanent_uuid: "c1b2193f640945a895e6497c6422849f" member_type: VOTER last_known_addr { host: "127.10.198.3" port: 41889 } } peers { permanent_uuid: "3cad46077ef143feae0d7ee5ac6852f4" member_type: VOTER last_known_addr { host: "127.10.198.1" port: 36901 } }
I20260925 22:44:30.227528 13269 ts_tablet_manager.cc:1428] T bf53324bfb8f4602b82b33b495239d3f P 2a585751d44c49f98359ac6423e39c71: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260925 22:44:30.228011 13106 heartbeater.cc:503] Master 127.10.198.62:42121 was elected leader, sending a full tablet report...
I20260925 22:44:30.227782 13270 raft_consensus.cc:348] T bf53324bfb8f4602b82b33b495239d3f P 3cad46077ef143feae0d7ee5ac6852f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a585751d44c49f98359ac6423e39c71" member_type: VOTER last_known_addr { host: "127.10.198.2" port: 45425 } } peers { permanent_uuid: "c1b2193f640945a895e6497c6422849f" member_type: VOTER last_known_addr { host: "127.10.198.3" port: 41889 } } peers { permanent_uuid: "3cad46077ef143feae0d7ee5ac6852f4" member_type: VOTER last_known_addr { host: "127.10.198.1" port: 36901 } }
I20260925 22:44:30.228127 13270 raft_consensus.cc:374] T bf53324bfb8f4602b82b33b495239d3f P 3cad46077ef143feae0d7ee5ac6852f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:44:30.228197 13270 raft_consensus.cc:729] T bf53324bfb8f4602b82b33b495239d3f P 3cad46077ef143feae0d7ee5ac6852f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3cad46077ef143feae0d7ee5ac6852f4, State: Initialized, Role: FOLLOWER
I20260925 22:44:30.228164 13271 raft_consensus.cc:348] T bf53324bfb8f4602b82b33b495239d3f P c1b2193f640945a895e6497c6422849f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a585751d44c49f98359ac6423e39c71" member_type: VOTER last_known_addr { host: "127.10.198.2" port: 45425 } } peers { permanent_uuid: "c1b2193f640945a895e6497c6422849f" member_type: VOTER last_known_addr { host: "127.10.198.3" port: 41889 } } peers { permanent_uuid: "3cad46077ef143feae0d7ee5ac6852f4" member_type: VOTER last_known_addr { host: "127.10.198.1" port: 36901 } }
I20260925 22:44:30.228416 13271 raft_consensus.cc:374] T bf53324bfb8f4602b82b33b495239d3f P c1b2193f640945a895e6497c6422849f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:44:30.228477 13271 raft_consensus.cc:729] T bf53324bfb8f4602b82b33b495239d3f P c1b2193f640945a895e6497c6422849f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c1b2193f640945a895e6497c6422849f, State: Initialized, Role: FOLLOWER
I20260925 22:44:30.228710 13270 consensus_queue.cc:260] T bf53324bfb8f4602b82b33b495239d3f P 3cad46077ef143feae0d7ee5ac6852f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a585751d44c49f98359ac6423e39c71" member_type: VOTER last_known_addr { host: "127.10.198.2" port: 45425 } } peers { permanent_uuid: "c1b2193f640945a895e6497c6422849f" member_type: VOTER last_known_addr { host: "127.10.198.3" port: 41889 } } peers { permanent_uuid: "3cad46077ef143feae0d7ee5ac6852f4" member_type: VOTER last_known_addr { host: "127.10.198.1" port: 36901 } }
I20260925 22:44:30.228905 13271 consensus_queue.cc:260] T bf53324bfb8f4602b82b33b495239d3f P c1b2193f640945a895e6497c6422849f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a585751d44c49f98359ac6423e39c71" member_type: VOTER last_known_addr { host: "127.10.198.2" port: 45425 } } peers { permanent_uuid: "c1b2193f640945a895e6497c6422849f" member_type: VOTER last_known_addr { host: "127.10.198.3" port: 41889 } } peers { permanent_uuid: "3cad46077ef143feae0d7ee5ac6852f4" member_type: VOTER last_known_addr { host: "127.10.198.1" port: 36901 } }
I20260925 22:44:30.229851 13246 heartbeater.cc:503] Master 127.10.198.62:42121 was elected leader, sending a full tablet report...
I20260925 22:44:30.229911 13271 ts_tablet_manager.cc:1428] T bf53324bfb8f4602b82b33b495239d3f P c1b2193f640945a895e6497c6422849f: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260925 22:44:30.229923 13270 ts_tablet_manager.cc:1428] T bf53324bfb8f4602b82b33b495239d3f P 3cad46077ef143feae0d7ee5ac6852f4: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260925 22:44:30.230415 12966 heartbeater.cc:503] Master 127.10.198.62:42121 was elected leader, sending a full tablet report...
I20260925 22:44:30.234510 13275 raft_consensus.cc:482] T bf53324bfb8f4602b82b33b495239d3f P 2a585751d44c49f98359ac6423e39c71 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 22:44:30.234741 13275 raft_consensus.cc:504] T bf53324bfb8f4602b82b33b495239d3f P 2a585751d44c49f98359ac6423e39c71 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a585751d44c49f98359ac6423e39c71" member_type: VOTER last_known_addr { host: "127.10.198.2" port: 45425 } } peers { permanent_uuid: "c1b2193f640945a895e6497c6422849f" member_type: VOTER last_known_addr { host: "127.10.198.3" port: 41889 } } peers { permanent_uuid: "3cad46077ef143feae0d7ee5ac6852f4" member_type: VOTER last_known_addr { host: "127.10.198.1" port: 36901 } }
I20260925 22:44:30.235955 13275 leader_election.cc:288] T bf53324bfb8f4602b82b33b495239d3f P 2a585751d44c49f98359ac6423e39c71 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c1b2193f640945a895e6497c6422849f (127.10.198.3:41889), 3cad46077ef143feae0d7ee5ac6852f4 (127.10.198.1:36901)
I20260925 22:44:30.239912 12999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:44:30.236151 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.198.3:41889 (local address 127.10.198.2:37203)
0925 22:44:30.236319 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0925 22:44:30.236337 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0925 22:44:30.236433 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:44:30.236762 (+ 329us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:44:30.236766 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:44:30.236786 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:44:30.236934 (+ 148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:44:30.236945 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:44:30.237762 (+ 817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:44:30.237768 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:44:30.238804 (+ 1036us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:44:30.238814 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:44:30.239472 (+ 658us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:44:30.239476 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:44:30.239583 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:44:30.239602 (+ 19us) client_negotiation.cc:769] Sending connection context
0925 22:44:30.239661 (+ 59us) client_negotiation.cc:240] Negotiation successful
0925 22:44:30.239723 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":81,"mutex_wait_us":27}
I20260925 22:44:30.239943 13267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:44:30.236446 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.198.2:37203 (local address 127.10.198.3:41889)
0925 22:44:30.236550 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0925 22:44:30.236554 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:44:30.236575 (+ 21us) server_negotiation.cc:407] Connection header received
0925 22:44:30.236619 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:44:30.236621 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:44:30.236658 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:44:30.236720 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:44:30.237044 (+ 324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:44:30.237648 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:44:30.238941 (+ 1293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:44:30.239350 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:44:30.239476 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:44:30.239532 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0925 22:44:30.239697 (+ 165us) server_negotiation.cc:299] Negotiation successful
0925 22:44:30.239782 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260925 22:44:30.240746 13278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:44:30.236151 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.198.1:36901 (local address 127.10.198.2:38435)
0925 22:44:30.236780 (+ 629us) negotiation.cc:107] Waiting for socket to connect
0925 22:44:30.236792 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 22:44:30.236883 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:44:30.237126 (+ 243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:44:30.237128 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:44:30.237142 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:44:30.237691 (+ 549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:44:30.237699 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:44:30.238437 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:44:30.238441 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:44:30.239362 (+ 921us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:44:30.239370 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:44:30.240324 (+ 954us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:44:30.240328 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:44:30.240432 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:44:30.240459 (+ 27us) client_negotiation.cc:769] Sending connection context
0925 22:44:30.240539 (+ 80us) client_negotiation.cc:240] Negotiation successful
0925 22:44:30.240615 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":568,"mutex_wait_us":38,"thread_start_us":140,"threads_started":1}
I20260925 22:44:30.240785 13268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:44:30.236386 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.198.2:38435 (local address 127.10.198.1:36901)
0925 22:44:30.236555 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0925 22:44:30.236559 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:44:30.236878 (+ 319us) server_negotiation.cc:407] Connection header received
0925 22:44:30.236965 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:44:30.236967 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:44:30.237034 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:44:30.237103 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:44:30.237843 (+ 740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:44:30.238321 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:44:30.239510 (+ 1189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:44:30.240172 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:44:30.240307 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:44:30.240355 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0925 22:44:30.240570 (+ 215us) server_negotiation.cc:299] Negotiation successful
0925 22:44:30.240675 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":98}
I20260925 22:44:30.240954 13200 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf53324bfb8f4602b82b33b495239d3f" candidate_uuid: "2a585751d44c49f98359ac6423e39c71" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c1b2193f640945a895e6497c6422849f" is_pre_election: true
I20260925 22:44:30.241315 13200 raft_consensus.cc:2454] T bf53324bfb8f4602b82b33b495239d3f P c1b2193f640945a895e6497c6422849f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2a585751d44c49f98359ac6423e39c71 in term 0.
I20260925 22:44:30.241359 12920 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf53324bfb8f4602b82b33b495239d3f" candidate_uuid: "2a585751d44c49f98359ac6423e39c71" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3cad46077ef143feae0d7ee5ac6852f4" is_pre_election: true
I20260925 22:44:30.241678 12920 raft_consensus.cc:2454] T bf53324bfb8f4602b82b33b495239d3f P 3cad46077ef143feae0d7ee5ac6852f4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2a585751d44c49f98359ac6423e39c71 in term 0.
I20260925 22:44:30.241914 12991 leader_election.cc:302] T bf53324bfb8f4602b82b33b495239d3f P 2a585751d44c49f98359ac6423e39c71 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2a585751d44c49f98359ac6423e39c71, c1b2193f640945a895e6497c6422849f; no voters:
I20260925 22:44:30.242249 13275 raft_consensus.cc:2781] T bf53324bfb8f4602b82b33b495239d3f P 2a585751d44c49f98359ac6423e39c71 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 22:44:30.242336 13275 raft_consensus.cc:482] T bf53324bfb8f4602b82b33b495239d3f P 2a585751d44c49f98359ac6423e39c71 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 22:44:30.242400 13275 raft_consensus.cc:3037] T bf53324bfb8f4602b82b33b495239d3f P 2a585751d44c49f98359ac6423e39c71 [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:44:30.243520 13275 raft_consensus.cc:504] T bf53324bfb8f4602b82b33b495239d3f P 2a585751d44c49f98359ac6423e39c71 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a585751d44c49f98359ac6423e39c71" member_type: VOTER last_known_addr { host: "127.10.198.2" port: 45425 } } peers { permanent_uuid: "c1b2193f640945a895e6497c6422849f" member_type: VOTER last_known_addr { host: "127.10.198.3" port: 41889 } } peers { permanent_uuid: "3cad46077ef143feae0d7ee5ac6852f4" member_type: VOTER last_known_addr { host: "127.10.198.1" port: 36901 } }
I20260925 22:44:30.243975 13275 leader_election.cc:288] T bf53324bfb8f4602b82b33b495239d3f P 2a585751d44c49f98359ac6423e39c71 [CANDIDATE]: Term 1 election: Requested vote from peers c1b2193f640945a895e6497c6422849f (127.10.198.3:41889), 3cad46077ef143feae0d7ee5ac6852f4 (127.10.198.1:36901)
I20260925 22:44:30.244342 13200 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf53324bfb8f4602b82b33b495239d3f" candidate_uuid: "2a585751d44c49f98359ac6423e39c71" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c1b2193f640945a895e6497c6422849f"
I20260925 22:44:30.244414 12920 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf53324bfb8f4602b82b33b495239d3f" candidate_uuid: "2a585751d44c49f98359ac6423e39c71" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3cad46077ef143feae0d7ee5ac6852f4"
I20260925 22:44:30.244500 13200 raft_consensus.cc:3037] T bf53324bfb8f4602b82b33b495239d3f P c1b2193f640945a895e6497c6422849f [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:44:30.244565 12920 raft_consensus.cc:3037] T bf53324bfb8f4602b82b33b495239d3f P 3cad46077ef143feae0d7ee5ac6852f4 [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:44:30.245666 12920 raft_consensus.cc:2454] T bf53324bfb8f4602b82b33b495239d3f P 3cad46077ef143feae0d7ee5ac6852f4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2a585751d44c49f98359ac6423e39c71 in term 1.
I20260925 22:44:30.245666 13200 raft_consensus.cc:2454] T bf53324bfb8f4602b82b33b495239d3f P c1b2193f640945a895e6497c6422849f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2a585751d44c49f98359ac6423e39c71 in term 1.
I20260925 22:44:30.246073 12993 leader_election.cc:302] T bf53324bfb8f4602b82b33b495239d3f P 2a585751d44c49f98359ac6423e39c71 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2a585751d44c49f98359ac6423e39c71, 3cad46077ef143feae0d7ee5ac6852f4; no voters:
I20260925 22:44:30.246320 13275 raft_consensus.cc:2781] T bf53324bfb8f4602b82b33b495239d3f P 2a585751d44c49f98359ac6423e39c71 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 22:44:30.246598 13275 raft_consensus.cc:686] T bf53324bfb8f4602b82b33b495239d3f P 2a585751d44c49f98359ac6423e39c71 [term 1 LEADER]: Becoming Leader. State: Replica: 2a585751d44c49f98359ac6423e39c71, State: Running, Role: LEADER
I20260925 22:44:30.246997 13275 consensus_queue.cc:237] T bf53324bfb8f4602b82b33b495239d3f P 2a585751d44c49f98359ac6423e39c71 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a585751d44c49f98359ac6423e39c71" member_type: VOTER last_known_addr { host: "127.10.198.2" port: 45425 } } peers { permanent_uuid: "c1b2193f640945a895e6497c6422849f" member_type: VOTER last_known_addr { host: "127.10.198.3" port: 41889 } } peers { permanent_uuid: "3cad46077ef143feae0d7ee5ac6852f4" member_type: VOTER last_known_addr { host: "127.10.198.1" port: 36901 } }
I20260925 22:44:30.250852 12762 catalog_manager.cc:5495] T bf53324bfb8f4602b82b33b495239d3f P 2a585751d44c49f98359ac6423e39c71 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2a585751d44c49f98359ac6423e39c71 (127.10.198.2). New cstate: current_term: 1 leader_uuid: "2a585751d44c49f98359ac6423e39c71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a585751d44c49f98359ac6423e39c71" member_type: VOTER last_known_addr { host: "127.10.198.2" port: 45425 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c1b2193f640945a895e6497c6422849f" member_type: VOTER last_known_addr { host: "127.10.198.3" port: 41889 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3cad46077ef143feae0d7ee5ac6852f4" member_type: VOTER last_known_addr { host: "127.10.198.1" port: 36901 } health_report { overall_health: UNKNOWN } } }
Sep 25 22:44:30 dist-test-slave-5vdr krb5kdc[12718](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376269, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.198.62@KRBTEST.COM for kudu/127.10.198.62@KRBTEST.COM
I20260925 22:44:30.291061 13286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:44:30.281770 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.198.62:42121 (local address 127.0.0.1:42288)
0925 22:44:30.282159 (+ 389us) negotiation.cc:107] Waiting for socket to connect
0925 22:44:30.282173 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 22:44:30.282345 (+ 172us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:44:30.282621 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:44:30.282623 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:44:30.283095 (+ 472us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:44:30.283193 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:44:30.283200 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:44:30.284067 (+ 867us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:44:30.284072 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:44:30.284644 (+ 572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:44:30.284653 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:44:30.285120 (+ 467us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:44:30.285123 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:44:30.285193 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:44:30.285940 (+ 747us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:44:30.285954 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:44:30.287454 (+ 1500us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:44:30.289313 (+ 1859us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:44:30.289323 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:44:30.289330 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:44:30.289577 (+ 247us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:44:30.289884 (+ 307us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:44:30.289887 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:44:30.289889 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:44:30.289971 (+ 82us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:44:30.290371 (+ 400us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:44:30.290379 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:44:30.290499 (+ 120us) client_negotiation.cc:769] Sending connection context
0925 22:44:30.290616 (+ 117us) client_negotiation.cc:240] Negotiation successful
0925 22:44:30.290821 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":315,"thread_start_us":146,"threads_started":1}
I20260925 22:44:30.291061 12812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:44:30.281804 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42288 (local address 127.10.198.62:42121)
0925 22:44:30.282009 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0925 22:44:30.282014 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:44:30.282284 (+ 270us) server_negotiation.cc:407] Connection header received
0925 22:44:30.282456 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:44:30.282460 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:44:30.282507 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:44:30.282597 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:44:30.283345 (+ 748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:44:30.283957 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:44:30.284778 (+ 821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:44:30.284998 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:44:30.285114 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:44:30.287630 (+ 2516us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:44:30.287634 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:44:30.287636 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:44:30.287659 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:44:30.289157 (+ 1498us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:44:30.289688 (+ 531us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:44:30.289693 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:44:30.289695 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:44:30.289783 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:44:30.290071 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:44:30.290074 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:44:30.290076 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:44:30.290262 (+ 186us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:44:30.290371 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0925 22:44:30.290673 (+ 302us) server_negotiation.cc:299] Negotiation successful
0925 22:44:30.290886 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":105,"server-negotiator.queue_time_us":79}
I20260925 22:44:30.295966 12762 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.10.198.62@KRBTEST.COM'} at 127.0.0.1:42288:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260925 22:44:30.296883 12762 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 22:44:30.303129 12899 tablet_service.cc:1467] Processing CreateTablet for tablet 2f7f69df34404106a4f591e1c0543e3d (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=161f652c3ee043daa39ae1075b887b1e]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260925 22:44:30.303186 13040 tablet_service.cc:1467] Processing CreateTablet for tablet 2f7f69df34404106a4f591e1c0543e3d (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=161f652c3ee043daa39ae1075b887b1e]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260925 22:44:30.304015 13040 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2f7f69df34404106a4f591e1c0543e3d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:44:30.304015 12899 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2f7f69df34404106a4f591e1c0543e3d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:44:30.303812 13180 tablet_service.cc:1467] Processing CreateTablet for tablet 2f7f69df34404106a4f591e1c0543e3d (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=161f652c3ee043daa39ae1075b887b1e]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260925 22:44:30.304571 13180 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2f7f69df34404106a4f591e1c0543e3d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:44:30.307178 13271 tablet_bootstrap.cc:492] T 2f7f69df34404106a4f591e1c0543e3d P c1b2193f640945a895e6497c6422849f: Bootstrap starting.
I20260925 22:44:30.307739 13269 tablet_bootstrap.cc:492] T 2f7f69df34404106a4f591e1c0543e3d P 2a585751d44c49f98359ac6423e39c71: Bootstrap starting.
I20260925 22:44:30.308434 13271 tablet_bootstrap.cc:654] T 2f7f69df34404106a4f591e1c0543e3d P c1b2193f640945a895e6497c6422849f: Neither blocks nor log segments found. Creating new log.
I20260925 22:44:30.308765 13270 tablet_bootstrap.cc:492] T 2f7f69df34404106a4f591e1c0543e3d P 3cad46077ef143feae0d7ee5ac6852f4: Bootstrap starting.
I20260925 22:44:30.308876 13269 tablet_bootstrap.cc:654] T 2f7f69df34404106a4f591e1c0543e3d P 2a585751d44c49f98359ac6423e39c71: Neither blocks nor log segments found. Creating new log.
I20260925 22:44:30.309770 13270 tablet_bootstrap.cc:654] T 2f7f69df34404106a4f591e1c0543e3d P 3cad46077ef143feae0d7ee5ac6852f4: Neither blocks nor log segments found. Creating new log.
I20260925 22:44:30.310048 13271 tablet_bootstrap.cc:492] T 2f7f69df34404106a4f591e1c0543e3d P c1b2193f640945a895e6497c6422849f: No bootstrap required, opened a new log
I20260925 22:44:30.310163 13271 ts_tablet_manager.cc:1397] T 2f7f69df34404106a4f591e1c0543e3d P c1b2193f640945a895e6497c6422849f: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260925 22:44:30.310233 13269 tablet_bootstrap.cc:492] T 2f7f69df34404106a4f591e1c0543e3d P 2a585751d44c49f98359ac6423e39c71: No bootstrap required, opened a new log
I20260925 22:44:30.310334 13269 ts_tablet_manager.cc:1397] T 2f7f69df34404106a4f591e1c0543e3d P 2a585751d44c49f98359ac6423e39c71: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260925 22:44:30.310683 13271 raft_consensus.cc:348] T 2f7f69df34404106a4f591e1c0543e3d P c1b2193f640945a895e6497c6422849f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cad46077ef143feae0d7ee5ac6852f4" member_type: VOTER last_known_addr { host: "127.10.198.1" port: 36901 } } peers { permanent_uuid: "2a585751d44c49f98359ac6423e39c71" member_type: VOTER last_known_addr { host: "127.10.198.2" port: 45425 } } peers { permanent_uuid: "c1b2193f640945a895e6497c6422849f" member_type: VOTER last_known_addr { host: "127.10.198.3" port: 41889 } }
I20260925 22:44:30.310848 13271 raft_consensus.cc:374] T 2f7f69df34404106a4f591e1c0543e3d P c1b2193f640945a895e6497c6422849f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:44:30.310806 13269 raft_consensus.cc:348] T 2f7f69df34404106a4f591e1c0543e3d P 2a585751d44c49f98359ac6423e39c71 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cad46077ef143feae0d7ee5ac6852f4" member_type: VOTER last_known_addr { host: "127.10.198.1" port: 36901 } } peers { permanent_uuid: "2a585751d44c49f98359ac6423e39c71" member_type: VOTER last_known_addr { host: "127.10.198.2" port: 45425 } } peers { permanent_uuid: "c1b2193f640945a895e6497c6422849f" member_type: VOTER last_known_addr { host: "127.10.198.3" port: 41889 } }
I20260925 22:44:30.310901 13271 raft_consensus.cc:729] T 2f7f69df34404106a4f591e1c0543e3d P c1b2193f640945a895e6497c6422849f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c1b2193f640945a895e6497c6422849f, State: Initialized, Role: FOLLOWER
I20260925 22:44:30.310926 13269 raft_consensus.cc:374] T 2f7f69df34404106a4f591e1c0543e3d P 2a585751d44c49f98359ac6423e39c71 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:44:30.310961 13269 raft_consensus.cc:729] T 2f7f69df34404106a4f591e1c0543e3d P 2a585751d44c49f98359ac6423e39c71 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2a585751d44c49f98359ac6423e39c71, State: Initialized, Role: FOLLOWER
I20260925 22:44:30.311081 13271 consensus_queue.cc:260] T 2f7f69df34404106a4f591e1c0543e3d P c1b2193f640945a895e6497c6422849f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cad46077ef143feae0d7ee5ac6852f4" member_type: VOTER last_known_addr { host: "127.10.198.1" port: 36901 } } peers { permanent_uuid: "2a585751d44c49f98359ac6423e39c71" member_type: VOTER last_known_addr { host: "127.10.198.2" port: 45425 } } peers { permanent_uuid: "c1b2193f640945a895e6497c6422849f" member_type: VOTER last_known_addr { host: "127.10.198.3" port: 41889 } }
I20260925 22:44:30.311137 13269 consensus_queue.cc:260] T 2f7f69df34404106a4f591e1c0543e3d P 2a585751d44c49f98359ac6423e39c71 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cad46077ef143feae0d7ee5ac6852f4" member_type: VOTER last_known_addr { host: "127.10.198.1" port: 36901 } } peers { permanent_uuid: "2a585751d44c49f98359ac6423e39c71" member_type: VOTER last_known_addr { host: "127.10.198.2" port: 45425 } } peers { permanent_uuid: "c1b2193f640945a895e6497c6422849f" member_type: VOTER last_known_addr { host: "127.10.198.3" port: 41889 } }
I20260925 22:44:30.311380 13271 ts_tablet_manager.cc:1428] T 2f7f69df34404106a4f591e1c0543e3d P c1b2193f640945a895e6497c6422849f: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20260925 22:44:30.311429 13269 ts_tablet_manager.cc:1428] T 2f7f69df34404106a4f591e1c0543e3d P 2a585751d44c49f98359ac6423e39c71: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260925 22:44:30.311587 13270 tablet_bootstrap.cc:492] T 2f7f69df34404106a4f591e1c0543e3d P 3cad46077ef143feae0d7ee5ac6852f4: No bootstrap required, opened a new log
I20260925 22:44:30.311685 13270 ts_tablet_manager.cc:1397] T 2f7f69df34404106a4f591e1c0543e3d P 3cad46077ef143feae0d7ee5ac6852f4: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260925 22:44:30.311640 13277 tablet_replica.cc:439] T 2f7f69df34404106a4f591e1c0543e3d P c1b2193f640945a895e6497c6422849f: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cad46077ef143feae0d7ee5ac6852f4" member_type: VOTER last_known_addr { host: "127.10.198.1" port: 36901 } } peers { permanent_uuid: "2a585751d44c49f98359ac6423e39c71" member_type: VOTER last_known_addr { host: "127.10.198.2" port: 45425 } } peers { permanent_uuid: "c1b2193f640945a895e6497c6422849f" member_type: VOTER last_known_addr { host: "127.10.198.3" port: 41889 } } }
I20260925 22:44:30.311685 13275 tablet_replica.cc:439] T 2f7f69df34404106a4f591e1c0543e3d P 2a585751d44c49f98359ac6423e39c71: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cad46077ef143feae0d7ee5ac6852f4" member_type: VOTER last_known_addr { host: "127.10.198.1" port: 36901 } } peers { permanent_uuid: "2a585751d44c49f98359ac6423e39c71" member_type: VOTER last_known_addr { host: "127.10.198.2" port: 45425 } } peers { permanent_uuid: "c1b2193f640945a895e6497c6422849f" member_type: VOTER last_known_addr { host: "127.10.198.3" port: 41889 } } }
I20260925 22:44:30.311792 13277 tablet_replica.cc:442] T 2f7f69df34404106a4f591e1c0543e3d P c1b2193f640945a895e6497c6422849f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 22:44:30.311802 13275 tablet_replica.cc:442] T 2f7f69df34404106a4f591e1c0543e3d P 2a585751d44c49f98359ac6423e39c71: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 22:44:30.312187 13270 raft_consensus.cc:348] T 2f7f69df34404106a4f591e1c0543e3d P 3cad46077ef143feae0d7ee5ac6852f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cad46077ef143feae0d7ee5ac6852f4" member_type: VOTER last_known_addr { host: "127.10.198.1" port: 36901 } } peers { permanent_uuid: "2a585751d44c49f98359ac6423e39c71" member_type: VOTER last_known_addr { host: "127.10.198.2" port: 45425 } } peers { permanent_uuid: "c1b2193f640945a895e6497c6422849f" member_type: VOTER last_known_addr { host: "127.10.198.3" port: 41889 } }
I20260925 22:44:30.312345 13270 raft_consensus.cc:374] T 2f7f69df34404106a4f591e1c0543e3d P 3cad46077ef143feae0d7ee5ac6852f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:44:30.312381 13270 raft_consensus.cc:729] T 2f7f69df34404106a4f591e1c0543e3d P 3cad46077ef143feae0d7ee5ac6852f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3cad46077ef143feae0d7ee5ac6852f4, State: Initialized, Role: FOLLOWER
I20260925 22:44:30.312494 13270 consensus_queue.cc:260] T 2f7f69df34404106a4f591e1c0543e3d P 3cad46077ef143feae0d7ee5ac6852f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cad46077ef143feae0d7ee5ac6852f4" member_type: VOTER last_known_addr { host: "127.10.198.1" port: 36901 } } peers { permanent_uuid: "2a585751d44c49f98359ac6423e39c71" member_type: VOTER last_known_addr { host: "127.10.198.2" port: 45425 } } peers { permanent_uuid: "c1b2193f640945a895e6497c6422849f" member_type: VOTER last_known_addr { host: "127.10.198.3" port: 41889 } }
I20260925 22:44:30.312762 13270 ts_tablet_manager.cc:1428] T 2f7f69df34404106a4f591e1c0543e3d P 3cad46077ef143feae0d7ee5ac6852f4: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260925 22:44:30.312825 13276 tablet_replica.cc:439] T 2f7f69df34404106a4f591e1c0543e3d P 3cad46077ef143feae0d7ee5ac6852f4: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cad46077ef143feae0d7ee5ac6852f4" member_type: VOTER last_known_addr { host: "127.10.198.1" port: 36901 } } peers { permanent_uuid: "2a585751d44c49f98359ac6423e39c71" member_type: VOTER last_known_addr { host: "127.10.198.2" port: 45425 } } peers { permanent_uuid: "c1b2193f640945a895e6497c6422849f" member_type: VOTER last_known_addr { host: "127.10.198.3" port: 41889 } } }
I20260925 22:44:30.312959 13276 tablet_replica.cc:442] T 2f7f69df34404106a4f591e1c0543e3d P 3cad46077ef143feae0d7ee5ac6852f4: This TxnStatusTablet replica's current role is: FOLLOWER
W20260925 22:44:30.380007 13247 tablet.cc:2369] T 2f7f69df34404106a4f591e1c0543e3d P c1b2193f640945a895e6497c6422849f: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 22:44:30.390372 13107 tablet.cc:2369] T 2f7f69df34404106a4f591e1c0543e3d P 2a585751d44c49f98359ac6423e39c71: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 22:44:30.398912 12967 tablet.cc:2369] T 2f7f69df34404106a4f591e1c0543e3d P 3cad46077ef143feae0d7ee5ac6852f4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 22:44:30.521678 13275 raft_consensus.cc:482] T 2f7f69df34404106a4f591e1c0543e3d P 2a585751d44c49f98359ac6423e39c71 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 22:44:30.521880 13275 raft_consensus.cc:504] T 2f7f69df34404106a4f591e1c0543e3d P 2a585751d44c49f98359ac6423e39c71 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cad46077ef143feae0d7ee5ac6852f4" member_type: VOTER last_known_addr { host: "127.10.198.1" port: 36901 } } peers { permanent_uuid: "2a585751d44c49f98359ac6423e39c71" member_type: VOTER last_known_addr { host: "127.10.198.2" port: 45425 } } peers { permanent_uuid: "c1b2193f640945a895e6497c6422849f" member_type: VOTER last_known_addr { host: "127.10.198.3" port: 41889 } }
I20260925 22:44:30.522348 13275 leader_election.cc:288] T 2f7f69df34404106a4f591e1c0543e3d P 2a585751d44c49f98359ac6423e39c71 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3cad46077ef143feae0d7ee5ac6852f4 (127.10.198.1:36901), c1b2193f640945a895e6497c6422849f (127.10.198.3:41889)
I20260925 22:44:30.522902 13200 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f7f69df34404106a4f591e1c0543e3d" candidate_uuid: "2a585751d44c49f98359ac6423e39c71" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c1b2193f640945a895e6497c6422849f" is_pre_election: true
I20260925 22:44:30.522912 12920 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f7f69df34404106a4f591e1c0543e3d" candidate_uuid: "2a585751d44c49f98359ac6423e39c71" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3cad46077ef143feae0d7ee5ac6852f4" is_pre_election: true
I20260925 22:44:30.523046 13200 raft_consensus.cc:2454] T 2f7f69df34404106a4f591e1c0543e3d P c1b2193f640945a895e6497c6422849f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2a585751d44c49f98359ac6423e39c71 in term 0.
I20260925 22:44:30.523120 12920 raft_consensus.cc:2454] T 2f7f69df34404106a4f591e1c0543e3d P 3cad46077ef143feae0d7ee5ac6852f4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2a585751d44c49f98359ac6423e39c71 in term 0.
I20260925 22:44:30.523480 12993 leader_election.cc:302] T 2f7f69df34404106a4f591e1c0543e3d P 2a585751d44c49f98359ac6423e39c71 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 2a585751d44c49f98359ac6423e39c71, 3cad46077ef143feae0d7ee5ac6852f4, c1b2193f640945a895e6497c6422849f; no voters:
I20260925 22:44:30.523715 13275 raft_consensus.cc:2781] T 2f7f69df34404106a4f591e1c0543e3d P 2a585751d44c49f98359ac6423e39c71 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 22:44:30.523780 13275 raft_consensus.cc:482] T 2f7f69df34404106a4f591e1c0543e3d P 2a585751d44c49f98359ac6423e39c71 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 22:44:30.523866 13275 raft_consensus.cc:3037] T 2f7f69df34404106a4f591e1c0543e3d P 2a585751d44c49f98359ac6423e39c71 [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:44:30.524868 13275 raft_consensus.cc:504] T 2f7f69df34404106a4f591e1c0543e3d P 2a585751d44c49f98359ac6423e39c71 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cad46077ef143feae0d7ee5ac6852f4" member_type: VOTER last_known_addr { host: "127.10.198.1" port: 36901 } } peers { permanent_uuid: "2a585751d44c49f98359ac6423e39c71" member_type: VOTER last_known_addr { host: "127.10.198.2" port: 45425 } } peers { permanent_uuid: "c1b2193f640945a895e6497c6422849f" member_type: VOTER last_known_addr { host: "127.10.198.3" port: 41889 } }
I20260925 22:44:30.525280 13275 leader_election.cc:288] T 2f7f69df34404106a4f591e1c0543e3d P 2a585751d44c49f98359ac6423e39c71 [CANDIDATE]: Term 1 election: Requested vote from peers 3cad46077ef143feae0d7ee5ac6852f4 (127.10.198.1:36901), c1b2193f640945a895e6497c6422849f (127.10.198.3:41889)
I20260925 22:44:30.525796 13200 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f7f69df34404106a4f591e1c0543e3d" candidate_uuid: "2a585751d44c49f98359ac6423e39c71" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c1b2193f640945a895e6497c6422849f"
I20260925 22:44:30.525959 13200 raft_consensus.cc:3037] T 2f7f69df34404106a4f591e1c0543e3d P c1b2193f640945a895e6497c6422849f [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:44:30.526185 12920 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f7f69df34404106a4f591e1c0543e3d" candidate_uuid: "2a585751d44c49f98359ac6423e39c71" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3cad46077ef143feae0d7ee5ac6852f4"
I20260925 22:44:30.526333 12920 raft_consensus.cc:3037] T 2f7f69df34404106a4f591e1c0543e3d P 3cad46077ef143feae0d7ee5ac6852f4 [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:44:30.527423 13200 raft_consensus.cc:2454] T 2f7f69df34404106a4f591e1c0543e3d P c1b2193f640945a895e6497c6422849f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2a585751d44c49f98359ac6423e39c71 in term 1.
I20260925 22:44:30.527604 12920 raft_consensus.cc:2454] T 2f7f69df34404106a4f591e1c0543e3d P 3cad46077ef143feae0d7ee5ac6852f4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2a585751d44c49f98359ac6423e39c71 in term 1.
I20260925 22:44:30.527917 12991 leader_election.cc:302] T 2f7f69df34404106a4f591e1c0543e3d P 2a585751d44c49f98359ac6423e39c71 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2a585751d44c49f98359ac6423e39c71, c1b2193f640945a895e6497c6422849f; no voters:
I20260925 22:44:30.528123 13275 raft_consensus.cc:2781] T 2f7f69df34404106a4f591e1c0543e3d P 2a585751d44c49f98359ac6423e39c71 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 22:44:30.528219 13275 raft_consensus.cc:686] T 2f7f69df34404106a4f591e1c0543e3d P 2a585751d44c49f98359ac6423e39c71 [term 1 LEADER]: Becoming Leader. State: Replica: 2a585751d44c49f98359ac6423e39c71, State: Running, Role: LEADER
I20260925 22:44:30.528394 13275 consensus_queue.cc:237] T 2f7f69df34404106a4f591e1c0543e3d P 2a585751d44c49f98359ac6423e39c71 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cad46077ef143feae0d7ee5ac6852f4" member_type: VOTER last_known_addr { host: "127.10.198.1" port: 36901 } } peers { permanent_uuid: "2a585751d44c49f98359ac6423e39c71" member_type: VOTER last_known_addr { host: "127.10.198.2" port: 45425 } } peers { permanent_uuid: "c1b2193f640945a895e6497c6422849f" member_type: VOTER last_known_addr { host: "127.10.198.3" port: 41889 } }
I20260925 22:44:30.529121 13279 tablet_replica.cc:439] T 2f7f69df34404106a4f591e1c0543e3d P 2a585751d44c49f98359ac6423e39c71: TxnStatusTablet state changed. Reason: New leader 2a585751d44c49f98359ac6423e39c71. Latest consensus state: current_term: 1 leader_uuid: "2a585751d44c49f98359ac6423e39c71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cad46077ef143feae0d7ee5ac6852f4" member_type: VOTER last_known_addr { host: "127.10.198.1" port: 36901 } } peers { permanent_uuid: "2a585751d44c49f98359ac6423e39c71" member_type: VOTER last_known_addr { host: "127.10.198.2" port: 45425 } } peers { permanent_uuid: "c1b2193f640945a895e6497c6422849f" member_type: VOTER last_known_addr { host: "127.10.198.3" port: 41889 } } }
I20260925 22:44:30.529248 13279 tablet_replica.cc:442] T 2f7f69df34404106a4f591e1c0543e3d P 2a585751d44c49f98359ac6423e39c71: This TxnStatusTablet replica's current role is: LEADER
I20260925 22:44:30.529520 13291 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260925 22:44:30.529762 13291 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260925 22:44:30.530349 12763 catalog_manager.cc:5495] T 2f7f69df34404106a4f591e1c0543e3d P 2a585751d44c49f98359ac6423e39c71 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2a585751d44c49f98359ac6423e39c71 (127.10.198.2). New cstate: current_term: 1 leader_uuid: "2a585751d44c49f98359ac6423e39c71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cad46077ef143feae0d7ee5ac6852f4" member_type: VOTER last_known_addr { host: "127.10.198.1" port: 36901 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2a585751d44c49f98359ac6423e39c71" member_type: VOTER last_known_addr { host: "127.10.198.2" port: 45425 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c1b2193f640945a895e6497c6422849f" member_type: VOTER last_known_addr { host: "127.10.198.3" port: 41889 } health_report { overall_health: UNKNOWN } } }
I20260925 22:44:30.561183 13264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:44:30.557044 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33708 (local address 127.10.198.2:45425)
0925 22:44:30.557183 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0925 22:44:30.557187 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:44:30.557204 (+ 17us) server_negotiation.cc:407] Connection header received
0925 22:44:30.557412 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:44:30.557415 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:44:30.557482 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:44:30.557600 (+ 118us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 22:44:30.557877 (+ 277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:44:30.558512 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:44:30.559485 (+ 973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:44:30.559720 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:44:30.559878 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:44:30.560246 (+ 368us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 22:44:30.560712 (+ 466us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 22:44:30.560860 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0925 22:44:30.560947 (+ 87us) server_negotiation.cc:299] Negotiation successful
0925 22:44:30.561013 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260925 22:44:30.561182 13286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:44:30.556907 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.198.2:45425 (local address 127.0.0.1:33708)
0925 22:44:30.557073 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0925 22:44:30.557092 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0925 22:44:30.557229 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:44:30.557618 (+ 389us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:44:30.557621 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:44:30.557629 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 22:44:30.557749 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:44:30.557755 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:44:30.558679 (+ 924us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:44:30.558684 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:44:30.559337 (+ 653us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:44:30.559349 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:44:30.559886 (+ 537us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:44:30.559890 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:44:30.559998 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:44:30.560056 (+ 58us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 22:44:30.560825 (+ 769us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 22:44:30.560848 (+ 23us) client_negotiation.cc:769] Sending connection context
0925 22:44:30.560928 (+ 80us) client_negotiation.cc:240] Negotiation successful
0925 22:44:30.561059 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":34}
I20260925 22:44:30.567504 13200 raft_consensus.cc:1261] T 2f7f69df34404106a4f591e1c0543e3d P c1b2193f640945a895e6497c6422849f [term 1 FOLLOWER]: Refusing update from remote peer 2a585751d44c49f98359ac6423e39c71: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 22:44:30.567521 12920 raft_consensus.cc:1261] T 2f7f69df34404106a4f591e1c0543e3d P 3cad46077ef143feae0d7ee5ac6852f4 [term 1 FOLLOWER]: Refusing update from remote peer 2a585751d44c49f98359ac6423e39c71: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 22:44:30.568449 13279 consensus_queue.cc:1035] T 2f7f69df34404106a4f591e1c0543e3d P 2a585751d44c49f98359ac6423e39c71 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3cad46077ef143feae0d7ee5ac6852f4" member_type: VOTER last_known_addr { host: "127.10.198.1" port: 36901 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 22:44:30.569012 13275 consensus_queue.cc:1035] T 2f7f69df34404106a4f591e1c0543e3d P 2a585751d44c49f98359ac6423e39c71 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c1b2193f640945a895e6497c6422849f" member_type: VOTER last_known_addr { host: "127.10.198.3" port: 41889 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 22:44:30.573012 13277 tablet_replica.cc:439] T 2f7f69df34404106a4f591e1c0543e3d P c1b2193f640945a895e6497c6422849f: TxnStatusTablet state changed. Reason: New leader 2a585751d44c49f98359ac6423e39c71. Latest consensus state: current_term: 1 leader_uuid: "2a585751d44c49f98359ac6423e39c71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cad46077ef143feae0d7ee5ac6852f4" member_type: VOTER last_known_addr { host: "127.10.198.1" port: 36901 } } peers { permanent_uuid: "2a585751d44c49f98359ac6423e39c71" member_type: VOTER last_known_addr { host: "127.10.198.2" port: 45425 } } peers { permanent_uuid: "c1b2193f640945a895e6497c6422849f" member_type: VOTER last_known_addr { host: "127.10.198.3" port: 41889 } } }
I20260925 22:44:30.573168 13277 tablet_replica.cc:442] T 2f7f69df34404106a4f591e1c0543e3d P c1b2193f640945a895e6497c6422849f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 22:44:30.573482 13276 tablet_replica.cc:439] T 2f7f69df34404106a4f591e1c0543e3d P 3cad46077ef143feae0d7ee5ac6852f4: TxnStatusTablet state changed. Reason: New leader 2a585751d44c49f98359ac6423e39c71. Latest consensus state: current_term: 1 leader_uuid: "2a585751d44c49f98359ac6423e39c71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cad46077ef143feae0d7ee5ac6852f4" member_type: VOTER last_known_addr { host: "127.10.198.1" port: 36901 } } peers { permanent_uuid: "2a585751d44c49f98359ac6423e39c71" member_type: VOTER last_known_addr { host: "127.10.198.2" port: 45425 } } peers { permanent_uuid: "c1b2193f640945a895e6497c6422849f" member_type: VOTER last_known_addr { host: "127.10.198.3" port: 41889 } } }
I20260925 22:44:30.573629 13276 tablet_replica.cc:442] T 2f7f69df34404106a4f591e1c0543e3d P 3cad46077ef143feae0d7ee5ac6852f4: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 22:44:30.577451 13275 tablet_replica.cc:439] T 2f7f69df34404106a4f591e1c0543e3d P 2a585751d44c49f98359ac6423e39c71: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "2a585751d44c49f98359ac6423e39c71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cad46077ef143feae0d7ee5ac6852f4" member_type: VOTER last_known_addr { host: "127.10.198.1" port: 36901 } } peers { permanent_uuid: "2a585751d44c49f98359ac6423e39c71" member_type: VOTER last_known_addr { host: "127.10.198.2" port: 45425 } } peers { permanent_uuid: "c1b2193f640945a895e6497c6422849f" member_type: VOTER last_known_addr { host: "127.10.198.3" port: 41889 } } }
I20260925 22:44:30.577613 13275 tablet_replica.cc:442] T 2f7f69df34404106a4f591e1c0543e3d P 2a585751d44c49f98359ac6423e39c71: This TxnStatusTablet replica's current role is: LEADER
I20260925 22:44:30.578474 13276 tablet_replica.cc:439] T 2f7f69df34404106a4f591e1c0543e3d P 3cad46077ef143feae0d7ee5ac6852f4: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "2a585751d44c49f98359ac6423e39c71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cad46077ef143feae0d7ee5ac6852f4" member_type: VOTER last_known_addr { host: "127.10.198.1" port: 36901 } } peers { permanent_uuid: "2a585751d44c49f98359ac6423e39c71" member_type: VOTER last_known_addr { host: "127.10.198.2" port: 45425 } } peers { permanent_uuid: "c1b2193f640945a895e6497c6422849f" member_type: VOTER last_known_addr { host: "127.10.198.3" port: 41889 } } }
I20260925 22:44:30.578608 13276 tablet_replica.cc:442] T 2f7f69df34404106a4f591e1c0543e3d P 3cad46077ef143feae0d7ee5ac6852f4: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 22:44:30.580336 13296 mvcc.cc:205] Tried to move back new op lower bound from 7333381204236324864 to 7333381204085952512. Current Snapshot: MvccSnapshot[applied={T|T < 7333381204236324864}]
I20260925 22:44:30.581310 13279 tablet_replica.cc:439] T 2f7f69df34404106a4f591e1c0543e3d P 2a585751d44c49f98359ac6423e39c71: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "2a585751d44c49f98359ac6423e39c71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cad46077ef143feae0d7ee5ac6852f4" member_type: VOTER last_known_addr { host: "127.10.198.1" port: 36901 } } peers { permanent_uuid: "2a585751d44c49f98359ac6423e39c71" member_type: VOTER last_known_addr { host: "127.10.198.2" port: 45425 } } peers { permanent_uuid: "c1b2193f640945a895e6497c6422849f" member_type: VOTER last_known_addr { host: "127.10.198.3" port: 41889 } } }
I20260925 22:44:30.581467 13279 tablet_replica.cc:442] T 2f7f69df34404106a4f591e1c0543e3d P 2a585751d44c49f98359ac6423e39c71: This TxnStatusTablet replica's current role is: LEADER
I20260925 22:44:30.582991 13277 tablet_replica.cc:439] T 2f7f69df34404106a4f591e1c0543e3d P c1b2193f640945a895e6497c6422849f: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "2a585751d44c49f98359ac6423e39c71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cad46077ef143feae0d7ee5ac6852f4" member_type: VOTER last_known_addr { host: "127.10.198.1" port: 36901 } } peers { permanent_uuid: "2a585751d44c49f98359ac6423e39c71" member_type: VOTER last_known_addr { host: "127.10.198.2" port: 45425 } } peers { permanent_uuid: "c1b2193f640945a895e6497c6422849f" member_type: VOTER last_known_addr { host: "127.10.198.3" port: 41889 } } }
I20260925 22:44:30.583138 13277 tablet_replica.cc:442] T 2f7f69df34404106a4f591e1c0543e3d P c1b2193f640945a895e6497c6422849f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 22:44:30.583360 13295 mvcc.cc:205] Tried to move back new op lower bound from 7333381204236324864 to 7333381204085952512. Current Snapshot: MvccSnapshot[applied={T|T < 7333381204236324864}]
I20260925 22:44:30.593770 13264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:44:30.589869 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33724 (local address 127.10.198.2:45425)
0925 22:44:30.589984 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0925 22:44:30.589988 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:44:30.590144 (+ 156us) server_negotiation.cc:407] Connection header received
0925 22:44:30.590289 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:44:30.590291 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:44:30.590337 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:44:30.590468 (+ 131us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 22:44:30.590728 (+ 260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:44:30.591230 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:44:30.592640 (+ 1410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:44:30.592879 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:44:30.592965 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:44:30.593322 (+ 357us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 22:44:30.593405 (+ 83us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 22:44:30.593466 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0925 22:44:30.593595 (+ 129us) server_negotiation.cc:299] Negotiation successful
0925 22:44:30.593649 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260925 22:44:30.606267 13264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:44:30.602169 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.198.2:51809 (local address 127.10.198.2:45425)
0925 22:44:30.602343 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0925 22:44:30.602348 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:44:30.602584 (+ 236us) server_negotiation.cc:407] Connection header received
0925 22:44:30.602755 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:44:30.602758 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:44:30.602805 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:44:30.602889 (+ 84us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 22:44:30.603118 (+ 229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:44:30.603715 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:44:30.605108 (+ 1393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:44:30.605321 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:44:30.605431 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:44:30.605773 (+ 342us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 22:44:30.605850 (+ 77us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 22:44:30.605937 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0925 22:44:30.606016 (+ 79us) server_negotiation.cc:299] Negotiation successful
0925 22:44:30.606090 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":38,"server-negotiator.queue_time_us":42}
I20260925 22:44:30.606277 13306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:44:30.602111 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.198.2:45425 (local address 127.10.198.2:51809)
0925 22:44:30.602490 (+ 379us) negotiation.cc:107] Waiting for socket to connect
0925 22:44:30.602504 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 22:44:30.602646 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:44:30.602895 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:44:30.602898 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:44:30.602904 (+ 6us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 22:44:30.603028 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:44:30.603036 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:44:30.603858 (+ 822us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:44:30.603864 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:44:30.604953 (+ 1089us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:44:30.604965 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:44:30.605437 (+ 472us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:44:30.605440 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:44:30.605559 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:44:30.605603 (+ 44us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 22:44:30.605937 (+ 334us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 22:44:30.605943 (+ 6us) client_negotiation.cc:769] Sending connection context
0925 22:44:30.606009 (+ 66us) client_negotiation.cc:240] Negotiation successful
0925 22:44:30.606111 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":285,"mutex_wait_us":13,"thread_start_us":132,"threads_started":1}
I20260925 22:44:30.614274 12920 raft_consensus.cc:1261] T bf53324bfb8f4602b82b33b495239d3f P 3cad46077ef143feae0d7ee5ac6852f4 [term 1 FOLLOWER]: Refusing update from remote peer 2a585751d44c49f98359ac6423e39c71: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 22:44:30.614306 13200 raft_consensus.cc:1261] T bf53324bfb8f4602b82b33b495239d3f P c1b2193f640945a895e6497c6422849f [term 1 FOLLOWER]: Refusing update from remote peer 2a585751d44c49f98359ac6423e39c71: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 22:44:30.614694 13301 consensus_queue.cc:1035] T bf53324bfb8f4602b82b33b495239d3f P 2a585751d44c49f98359ac6423e39c71 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3cad46077ef143feae0d7ee5ac6852f4" member_type: VOTER last_known_addr { host: "127.10.198.1" port: 36901 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 22:44:30.614864 13275 consensus_queue.cc:1035] T bf53324bfb8f4602b82b33b495239d3f P 2a585751d44c49f98359ac6423e39c71 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c1b2193f640945a895e6497c6422849f" member_type: VOTER last_known_addr { host: "127.10.198.3" port: 41889 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 22:44:30.618860 13294 mvcc.cc:205] Tried to move back new op lower bound from 7333381204432347136 to 7333381202934964224. Current Snapshot: MvccSnapshot[applied={T|T < 7333381204432347136}]
I20260925 22:44:30.672729 12763 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42286:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 22:44:30.672964 12763 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42286:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 22:44:30.676007 12763 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9f0e816c575a42caac240acda127e507: Sending DeleteTablet for 3 replicas of tablet bf53324bfb8f4602b82b33b495239d3f
I20260925 22:44:30.677173 13040 tablet_service.cc:1514] Processing DeleteTablet for tablet bf53324bfb8f4602b82b33b495239d3f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 22:44:30 UTC) from {username='kudu', principal='kudu/127.10.198.62@KRBTEST.COM'} at 127.0.0.1:33700
I20260925 22:44:30.677201 12899 tablet_service.cc:1514] Processing DeleteTablet for tablet bf53324bfb8f4602b82b33b495239d3f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 22:44:30 UTC) from {username='kudu', principal='kudu/127.10.198.62@KRBTEST.COM'} at 127.0.0.1:38708
I20260925 22:44:30.677299 13180 tablet_service.cc:1514] Processing DeleteTablet for tablet bf53324bfb8f4602b82b33b495239d3f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 22:44:30 UTC) from {username='kudu', principal='kudu/127.10.198.62@KRBTEST.COM'} at 127.0.0.1:45840
I20260925 22:44:30.677819 13313 tablet_replica.cc:330] T bf53324bfb8f4602b82b33b495239d3f P 3cad46077ef143feae0d7ee5ac6852f4: stopping tablet replica
I20260925 22:44:30.678094 13311 tablet_replica.cc:330] T bf53324bfb8f4602b82b33b495239d3f P c1b2193f640945a895e6497c6422849f: stopping tablet replica
I20260925 22:44:30.678246 13313 raft_consensus.cc:2229] T bf53324bfb8f4602b82b33b495239d3f P 3cad46077ef143feae0d7ee5ac6852f4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 22:44:30.678506 13313 raft_consensus.cc:2258] T bf53324bfb8f4602b82b33b495239d3f P 3cad46077ef143feae0d7ee5ac6852f4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 22:44:30.678453 13311 raft_consensus.cc:2229] T bf53324bfb8f4602b82b33b495239d3f P c1b2193f640945a895e6497c6422849f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 22:44:30.679174 13311 raft_consensus.cc:2258] T bf53324bfb8f4602b82b33b495239d3f P c1b2193f640945a895e6497c6422849f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 22:44:30.679173 12763 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42286:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 22:44:30.679769 12763 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 22:44:30.680229 13313 ts_tablet_manager.cc:1905] T bf53324bfb8f4602b82b33b495239d3f P 3cad46077ef143feae0d7ee5ac6852f4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 22:44:30.680872 13311 ts_tablet_manager.cc:1905] T bf53324bfb8f4602b82b33b495239d3f P c1b2193f640945a895e6497c6422849f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 22:44:30.684528 13311 ts_tablet_manager.cc:1918] T bf53324bfb8f4602b82b33b495239d3f P c1b2193f640945a895e6497c6422849f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 22:44:30.684556 13313 ts_tablet_manager.cc:1918] T bf53324bfb8f4602b82b33b495239d3f P 3cad46077ef143feae0d7ee5ac6852f4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 22:44:30.684670 13311 log.cc:1196] T bf53324bfb8f4602b82b33b495239d3f P c1b2193f640945a895e6497c6422849f: Deleting WAL directory at /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790376259650642-11032-0/minicluster-data/ts-2/wal/wals/bf53324bfb8f4602b82b33b495239d3f
I20260925 22:44:30.684674 13313 log.cc:1196] T bf53324bfb8f4602b82b33b495239d3f P 3cad46077ef143feae0d7ee5ac6852f4: Deleting WAL directory at /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790376259650642-11032-0/minicluster-data/ts-0/wal/wals/bf53324bfb8f4602b82b33b495239d3f
I20260925 22:44:30.685052 13311 ts_tablet_manager.cc:1939] T bf53324bfb8f4602b82b33b495239d3f P c1b2193f640945a895e6497c6422849f: Deleting consensus metadata
I20260925 22:44:30.685087 13313 ts_tablet_manager.cc:1939] T bf53324bfb8f4602b82b33b495239d3f P 3cad46077ef143feae0d7ee5ac6852f4: Deleting consensus metadata
I20260925 22:44:30.686694 13312 tablet_replica.cc:330] T bf53324bfb8f4602b82b33b495239d3f P 2a585751d44c49f98359ac6423e39c71: stopping tablet replica
I20260925 22:44:30.686735 12749 catalog_manager.cc:4841] TS 3cad46077ef143feae0d7ee5ac6852f4 (127.10.198.1:36901): tablet bf53324bfb8f4602b82b33b495239d3f (table test-table [id=c5a50e0d9a7e47cd9e897abd2ea9787f]) successfully deleted
I20260925 22:44:30.687093 13312 raft_consensus.cc:2229] T bf53324bfb8f4602b82b33b495239d3f P 2a585751d44c49f98359ac6423e39c71 [term 1 LEADER]: Raft consensus shutting down.
I20260925 22:44:30.687496 12899 tablet_service.cc:1467] Processing CreateTablet for tablet aa29a660ca9044f3b387a685651cdc7f (DEFAULT_TABLE table=test-table [id=f3a861f0125a4777be949439e8b0cba5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 22:44:30.687546 13312 raft_consensus.cc:2258] T bf53324bfb8f4602b82b33b495239d3f P 2a585751d44c49f98359ac6423e39c71 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 22:44:30.687628 13040 tablet_service.cc:1467] Processing CreateTablet for tablet aa29a660ca9044f3b387a685651cdc7f (DEFAULT_TABLE table=test-table [id=f3a861f0125a4777be949439e8b0cba5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 22:44:30.687826 12899 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aa29a660ca9044f3b387a685651cdc7f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:44:30.687966 13040 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aa29a660ca9044f3b387a685651cdc7f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:44:30.689059 12747 catalog_manager.cc:4841] TS c1b2193f640945a895e6497c6422849f (127.10.198.3:41889): tablet bf53324bfb8f4602b82b33b495239d3f (table test-table [id=c5a50e0d9a7e47cd9e897abd2ea9787f]) successfully deleted
I20260925 22:44:30.689679 13180 tablet_service.cc:1467] Processing CreateTablet for tablet aa29a660ca9044f3b387a685651cdc7f (DEFAULT_TABLE table=test-table [id=f3a861f0125a4777be949439e8b0cba5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 22:44:30.690001 13180 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aa29a660ca9044f3b387a685651cdc7f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:44:30.690651 13270 tablet_bootstrap.cc:492] T aa29a660ca9044f3b387a685651cdc7f P 3cad46077ef143feae0d7ee5ac6852f4: Bootstrap starting.
I20260925 22:44:30.690665 13269 tablet_bootstrap.cc:492] T aa29a660ca9044f3b387a685651cdc7f P 2a585751d44c49f98359ac6423e39c71: Bootstrap starting.
I20260925 22:44:30.691697 13270 tablet_bootstrap.cc:654] T aa29a660ca9044f3b387a685651cdc7f P 3cad46077ef143feae0d7ee5ac6852f4: Neither blocks nor log segments found. Creating new log.
I20260925 22:44:30.691793 13269 tablet_bootstrap.cc:654] T aa29a660ca9044f3b387a685651cdc7f P 2a585751d44c49f98359ac6423e39c71: Neither blocks nor log segments found. Creating new log.
I20260925 22:44:30.693045 13271 tablet_bootstrap.cc:492] T aa29a660ca9044f3b387a685651cdc7f P c1b2193f640945a895e6497c6422849f: Bootstrap starting.
I20260925 22:44:30.694300 13271 tablet_bootstrap.cc:654] T aa29a660ca9044f3b387a685651cdc7f P c1b2193f640945a895e6497c6422849f: Neither blocks nor log segments found. Creating new log.
I20260925 22:44:30.694281 13312 ts_tablet_manager.cc:1905] T bf53324bfb8f4602b82b33b495239d3f P 2a585751d44c49f98359ac6423e39c71: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 22:44:30.695953 13269 tablet_bootstrap.cc:492] T aa29a660ca9044f3b387a685651cdc7f P 2a585751d44c49f98359ac6423e39c71: No bootstrap required, opened a new log
I20260925 22:44:30.695984 13271 tablet_bootstrap.cc:492] T aa29a660ca9044f3b387a685651cdc7f P c1b2193f640945a895e6497c6422849f: No bootstrap required, opened a new log
I20260925 22:44:30.696081 13269 ts_tablet_manager.cc:1397] T aa29a660ca9044f3b387a685651cdc7f P 2a585751d44c49f98359ac6423e39c71: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20260925 22:44:30.696098 13271 ts_tablet_manager.cc:1397] T aa29a660ca9044f3b387a685651cdc7f P c1b2193f640945a895e6497c6422849f: Time spent bootstrapping tablet: real 0.003s user 0.003s sys 0.000s
I20260925 22:44:30.696341 13270 tablet_bootstrap.cc:492] T aa29a660ca9044f3b387a685651cdc7f P 3cad46077ef143feae0d7ee5ac6852f4: No bootstrap required, opened a new log
I20260925 22:44:30.696447 13270 ts_tablet_manager.cc:1397] T aa29a660ca9044f3b387a685651cdc7f P 3cad46077ef143feae0d7ee5ac6852f4: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.002s
I20260925 22:44:30.696678 13269 raft_consensus.cc:348] T aa29a660ca9044f3b387a685651cdc7f P 2a585751d44c49f98359ac6423e39c71 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cad46077ef143feae0d7ee5ac6852f4" member_type: VOTER last_known_addr { host: "127.10.198.1" port: 36901 } } peers { permanent_uuid: "2a585751d44c49f98359ac6423e39c71" member_type: VOTER last_known_addr { host: "127.10.198.2" port: 45425 } } peers { permanent_uuid: "c1b2193f640945a895e6497c6422849f" member_type: VOTER last_known_addr { host: "127.10.198.3" port: 41889 } }
I20260925 22:44:30.696841 13269 raft_consensus.cc:374] T aa29a660ca9044f3b387a685651cdc7f P 2a585751d44c49f98359ac6423e39c71 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:44:30.696892 13269 raft_consensus.cc:729] T aa29a660ca9044f3b387a685651cdc7f P 2a585751d44c49f98359ac6423e39c71 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2a585751d44c49f98359ac6423e39c71, State: Initialized, Role: FOLLOWER
I20260925 22:44:30.697052 13269 consensus_queue.cc:260] T aa29a660ca9044f3b387a685651cdc7f P 2a585751d44c49f98359ac6423e39c71 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cad46077ef143feae0d7ee5ac6852f4" member_type: VOTER last_known_addr { host: "127.10.198.1" port: 36901 } } peers { permanent_uuid: "2a585751d44c49f98359ac6423e39c71" member_type: VOTER last_known_addr { host: "127.10.198.2" port: 45425 } } peers { permanent_uuid: "c1b2193f640945a895e6497c6422849f" member_type: VOTER last_known_addr { host: "127.10.198.3" port: 41889 } }
I20260925 22:44:30.697017 13270 raft_consensus.cc:348] T aa29a660ca9044f3b387a685651cdc7f P 3cad46077ef143feae0d7ee5ac6852f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cad46077ef143feae0d7ee5ac6852f4" member_type: VOTER last_known_addr { host: "127.10.198.1" port: 36901 } } peers { permanent_uuid: "2a585751d44c49f98359ac6423e39c71" member_type: VOTER last_known_addr { host: "127.10.198.2" port: 45425 } } peers { permanent_uuid: "c1b2193f640945a895e6497c6422849f" member_type: VOTER last_known_addr { host: "127.10.198.3" port: 41889 } }
I20260925 22:44:30.697163 13270 raft_consensus.cc:374] T aa29a660ca9044f3b387a685651cdc7f P 3cad46077ef143feae0d7ee5ac6852f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:44:30.697220 13270 raft_consensus.cc:729] T aa29a660ca9044f3b387a685651cdc7f P 3cad46077ef143feae0d7ee5ac6852f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3cad46077ef143feae0d7ee5ac6852f4, State: Initialized, Role: FOLLOWER
I20260925 22:44:30.697345 13269 ts_tablet_manager.cc:1428] T aa29a660ca9044f3b387a685651cdc7f P 2a585751d44c49f98359ac6423e39c71: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260925 22:44:30.697393 13270 consensus_queue.cc:260] T aa29a660ca9044f3b387a685651cdc7f P 3cad46077ef143feae0d7ee5ac6852f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cad46077ef143feae0d7ee5ac6852f4" member_type: VOTER last_known_addr { host: "127.10.198.1" port: 36901 } } peers { permanent_uuid: "2a585751d44c49f98359ac6423e39c71" member_type: VOTER last_known_addr { host: "127.10.198.2" port: 45425 } } peers { permanent_uuid: "c1b2193f640945a895e6497c6422849f" member_type: VOTER last_known_addr { host: "127.10.198.3" port: 41889 } }
I20260925 22:44:30.697719 13270 ts_tablet_manager.cc:1428] T aa29a660ca9044f3b387a685651cdc7f P 3cad46077ef143feae0d7ee5ac6852f4: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20260925 22:44:30.698089 13312 ts_tablet_manager.cc:1918] T bf53324bfb8f4602b82b33b495239d3f P 2a585751d44c49f98359ac6423e39c71: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 22:44:30.698251 13312 log.cc:1196] T bf53324bfb8f4602b82b33b495239d3f P 2a585751d44c49f98359ac6423e39c71: Deleting WAL directory at /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790376259650642-11032-0/minicluster-data/ts-1/wal/wals/bf53324bfb8f4602b82b33b495239d3f
I20260925 22:44:30.698654 13312 ts_tablet_manager.cc:1939] T bf53324bfb8f4602b82b33b495239d3f P 2a585751d44c49f98359ac6423e39c71: Deleting consensus metadata
I20260925 22:44:30.698935 13271 raft_consensus.cc:348] T aa29a660ca9044f3b387a685651cdc7f P c1b2193f640945a895e6497c6422849f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cad46077ef143feae0d7ee5ac6852f4" member_type: VOTER last_known_addr { host: "127.10.198.1" port: 36901 } } peers { permanent_uuid: "2a585751d44c49f98359ac6423e39c71" member_type: VOTER last_known_addr { host: "127.10.198.2" port: 45425 } } peers { permanent_uuid: "c1b2193f640945a895e6497c6422849f" member_type: VOTER last_known_addr { host: "127.10.198.3" port: 41889 } }
I20260925 22:44:30.699091 13271 raft_consensus.cc:374] T aa29a660ca9044f3b387a685651cdc7f P c1b2193f640945a895e6497c6422849f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:44:30.699133 13271 raft_consensus.cc:729] T aa29a660ca9044f3b387a685651cdc7f P c1b2193f640945a895e6497c6422849f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c1b2193f640945a895e6497c6422849f, State: Initialized, Role: FOLLOWER
I20260925 22:44:30.699316 13271 consensus_queue.cc:260] T aa29a660ca9044f3b387a685651cdc7f P c1b2193f640945a895e6497c6422849f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cad46077ef143feae0d7ee5ac6852f4" member_type: VOTER last_known_addr { host: "127.10.198.1" port: 36901 } } peers { permanent_uuid: "2a585751d44c49f98359ac6423e39c71" member_type: VOTER last_known_addr { host: "127.10.198.2" port: 45425 } } peers { permanent_uuid: "c1b2193f640945a895e6497c6422849f" member_type: VOTER last_known_addr { host: "127.10.198.3" port: 41889 } }
I20260925 22:44:30.699666 13271 ts_tablet_manager.cc:1428] T aa29a660ca9044f3b387a685651cdc7f P c1b2193f640945a895e6497c6422849f: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20260925 22:44:30.700666 12748 catalog_manager.cc:4841] TS 2a585751d44c49f98359ac6423e39c71 (127.10.198.2:45425): tablet bf53324bfb8f4602b82b33b495239d3f (table test-table [id=c5a50e0d9a7e47cd9e897abd2ea9787f]) successfully deleted
I20260925 22:44:30.729482 13277 raft_consensus.cc:482] T aa29a660ca9044f3b387a685651cdc7f P c1b2193f640945a895e6497c6422849f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 22:44:30.729628 13277 raft_consensus.cc:504] T aa29a660ca9044f3b387a685651cdc7f P c1b2193f640945a895e6497c6422849f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cad46077ef143feae0d7ee5ac6852f4" member_type: VOTER last_known_addr { host: "127.10.198.1" port: 36901 } } peers { permanent_uuid: "2a585751d44c49f98359ac6423e39c71" member_type: VOTER last_known_addr { host: "127.10.198.2" port: 45425 } } peers { permanent_uuid: "c1b2193f640945a895e6497c6422849f" member_type: VOTER last_known_addr { host: "127.10.198.3" port: 41889 } }
I20260925 22:44:30.730824 13277 leader_election.cc:288] T aa29a660ca9044f3b387a685651cdc7f P c1b2193f640945a895e6497c6422849f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3cad46077ef143feae0d7ee5ac6852f4 (127.10.198.1:36901), 2a585751d44c49f98359ac6423e39c71 (127.10.198.2:45425)
I20260925 22:44:30.735904 13264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:44:30.731423 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.198.3:46853 (local address 127.10.198.2:45425)
0925 22:44:30.731552 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0925 22:44:30.731559 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 22:44:30.731771 (+ 212us) server_negotiation.cc:407] Connection header received
0925 22:44:30.732039 (+ 268us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:44:30.732043 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:44:30.732096 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:44:30.732147 (+ 51us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:44:30.732675 (+ 528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:44:30.733472 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:44:30.734727 (+ 1255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:44:30.735221 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:44:30.735350 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:44:30.735413 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0925 22:44:30.735648 (+ 235us) server_negotiation.cc:299] Negotiation successful
0925 22:44:30.735727 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260925 22:44:30.735919 13318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:44:30.731112 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.198.2:45425 (local address 127.10.198.3:46853)
0925 22:44:30.731686 (+ 574us) negotiation.cc:107] Waiting for socket to connect
0925 22:44:30.731703 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 22:44:30.731900 (+ 197us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:44:30.732310 (+ 410us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:44:30.732313 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:44:30.732336 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:44:30.732497 (+ 161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:44:30.732506 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:44:30.733640 (+ 1134us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:44:30.733643 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:44:30.734586 (+ 943us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:44:30.734597 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:44:30.735358 (+ 761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:44:30.735362 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:44:30.735459 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:44:30.735487 (+ 28us) client_negotiation.cc:769] Sending connection context
0925 22:44:30.735610 (+ 123us) client_negotiation.cc:240] Negotiation successful
0925 22:44:30.735694 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":505,"mutex_wait_us":71,"thread_start_us":77,"threads_started":1}
I20260925 22:44:30.736644 13268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:44:30.731369 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.198.3:37891 (local address 127.10.198.1:36901)
0925 22:44:30.731551 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0925 22:44:30.731556 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:44:30.731884 (+ 328us) server_negotiation.cc:407] Connection header received
0925 22:44:30.732106 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:44:30.732110 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:44:30.732157 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:44:30.732221 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:44:30.732715 (+ 494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:44:30.733472 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:44:30.735141 (+ 1669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:44:30.735815 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:44:30.735977 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:44:30.736033 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0925 22:44:30.736417 (+ 384us) server_negotiation.cc:299] Negotiation successful
0925 22:44:30.736515 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260925 22:44:30.736734 13060 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa29a660ca9044f3b387a685651cdc7f" candidate_uuid: "c1b2193f640945a895e6497c6422849f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a585751d44c49f98359ac6423e39c71" is_pre_election: true
I20260925 22:44:30.737015 13060 raft_consensus.cc:2454] T aa29a660ca9044f3b387a685651cdc7f P 2a585751d44c49f98359ac6423e39c71 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c1b2193f640945a895e6497c6422849f in term 0.
I20260925 22:44:30.737092 13317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:44:30.731117 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.198.1:36901 (local address 127.10.198.3:37891)
0925 22:44:30.731758 (+ 641us) negotiation.cc:107] Waiting for socket to connect
0925 22:44:30.731773 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 22:44:30.731972 (+ 199us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:44:30.732363 (+ 391us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:44:30.732366 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:44:30.732382 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:44:30.732586 (+ 204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:44:30.732594 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:44:30.733633 (+ 1039us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:44:30.733639 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:44:30.734980 (+ 1341us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:44:30.734992 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:44:30.736123 (+ 1131us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:44:30.736127 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:44:30.736256 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:44:30.736281 (+ 25us) client_negotiation.cc:769] Sending connection context
0925 22:44:30.736867 (+ 586us) client_negotiation.cc:240] Negotiation successful
0925 22:44:30.736951 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":564,"mutex_wait_us":331,"thread_start_us":71,"threads_started":1}
I20260925 22:44:30.737478 13132 leader_election.cc:302] T aa29a660ca9044f3b387a685651cdc7f P c1b2193f640945a895e6497c6422849f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2a585751d44c49f98359ac6423e39c71, c1b2193f640945a895e6497c6422849f; no voters:
I20260925 22:44:30.737690 12919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa29a660ca9044f3b387a685651cdc7f" candidate_uuid: "c1b2193f640945a895e6497c6422849f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3cad46077ef143feae0d7ee5ac6852f4" is_pre_election: true
I20260925 22:44:30.737828 13277 raft_consensus.cc:2781] T aa29a660ca9044f3b387a685651cdc7f P c1b2193f640945a895e6497c6422849f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 22:44:30.737850 12919 raft_consensus.cc:2454] T aa29a660ca9044f3b387a685651cdc7f P 3cad46077ef143feae0d7ee5ac6852f4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c1b2193f640945a895e6497c6422849f in term 0.
I20260925 22:44:30.737888 13277 raft_consensus.cc:482] T aa29a660ca9044f3b387a685651cdc7f P c1b2193f640945a895e6497c6422849f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 22:44:30.737918 13277 raft_consensus.cc:3037] T aa29a660ca9044f3b387a685651cdc7f P c1b2193f640945a895e6497c6422849f [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:44:30.738977 13277 raft_consensus.cc:504] T aa29a660ca9044f3b387a685651cdc7f P c1b2193f640945a895e6497c6422849f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cad46077ef143feae0d7ee5ac6852f4" member_type: VOTER last_known_addr { host: "127.10.198.1" port: 36901 } } peers { permanent_uuid: "2a585751d44c49f98359ac6423e39c71" member_type: VOTER last_known_addr { host: "127.10.198.2" port: 45425 } } peers { permanent_uuid: "c1b2193f640945a895e6497c6422849f" member_type: VOTER last_known_addr { host: "127.10.198.3" port: 41889 } }
I20260925 22:44:30.739333 13277 leader_election.cc:288] T aa29a660ca9044f3b387a685651cdc7f P c1b2193f640945a895e6497c6422849f [CANDIDATE]: Term 1 election: Requested vote from peers 3cad46077ef143feae0d7ee5ac6852f4 (127.10.198.1:36901), 2a585751d44c49f98359ac6423e39c71 (127.10.198.2:45425)
I20260925 22:44:30.739701 13060 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa29a660ca9044f3b387a685651cdc7f" candidate_uuid: "c1b2193f640945a895e6497c6422849f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a585751d44c49f98359ac6423e39c71"
I20260925 22:44:30.739735 12919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa29a660ca9044f3b387a685651cdc7f" candidate_uuid: "c1b2193f640945a895e6497c6422849f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3cad46077ef143feae0d7ee5ac6852f4"
I20260925 22:44:30.739847 13060 raft_consensus.cc:3037] T aa29a660ca9044f3b387a685651cdc7f P 2a585751d44c49f98359ac6423e39c71 [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:44:30.739912 12919 raft_consensus.cc:3037] T aa29a660ca9044f3b387a685651cdc7f P 3cad46077ef143feae0d7ee5ac6852f4 [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:44:30.741192 13060 raft_consensus.cc:2454] T aa29a660ca9044f3b387a685651cdc7f P 2a585751d44c49f98359ac6423e39c71 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c1b2193f640945a895e6497c6422849f in term 1.
I20260925 22:44:30.741271 12919 raft_consensus.cc:2454] T aa29a660ca9044f3b387a685651cdc7f P 3cad46077ef143feae0d7ee5ac6852f4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c1b2193f640945a895e6497c6422849f in term 1.
I20260925 22:44:30.741573 13132 leader_election.cc:302] T aa29a660ca9044f3b387a685651cdc7f P c1b2193f640945a895e6497c6422849f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2a585751d44c49f98359ac6423e39c71, c1b2193f640945a895e6497c6422849f; no voters:
I20260925 22:44:30.741838 13277 raft_consensus.cc:2781] T aa29a660ca9044f3b387a685651cdc7f P c1b2193f640945a895e6497c6422849f [term 1 FOLLOWER]: Leader election won for term 1
I20260925 22:44:30.742059 13277 raft_consensus.cc:686] T aa29a660ca9044f3b387a685651cdc7f P c1b2193f640945a895e6497c6422849f [term 1 LEADER]: Becoming Leader. State: Replica: c1b2193f640945a895e6497c6422849f, State: Running, Role: LEADER
I20260925 22:44:30.742339 13277 consensus_queue.cc:237] T aa29a660ca9044f3b387a685651cdc7f P c1b2193f640945a895e6497c6422849f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cad46077ef143feae0d7ee5ac6852f4" member_type: VOTER last_known_addr { host: "127.10.198.1" port: 36901 } } peers { permanent_uuid: "2a585751d44c49f98359ac6423e39c71" member_type: VOTER last_known_addr { host: "127.10.198.2" port: 45425 } } peers { permanent_uuid: "c1b2193f640945a895e6497c6422849f" member_type: VOTER last_known_addr { host: "127.10.198.3" port: 41889 } }
I20260925 22:44:30.745227 12762 catalog_manager.cc:5495] T aa29a660ca9044f3b387a685651cdc7f P c1b2193f640945a895e6497c6422849f reported cstate change: term changed from 0 to 1, leader changed from <none> to c1b2193f640945a895e6497c6422849f (127.10.198.3). New cstate: current_term: 1 leader_uuid: "c1b2193f640945a895e6497c6422849f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cad46077ef143feae0d7ee5ac6852f4" member_type: VOTER last_known_addr { host: "127.10.198.1" port: 36901 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2a585751d44c49f98359ac6423e39c71" member_type: VOTER last_known_addr { host: "127.10.198.2" port: 45425 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c1b2193f640945a895e6497c6422849f" member_type: VOTER last_known_addr { host: "127.10.198.3" port: 41889 } health_report { overall_health: HEALTHY } } }
I20260925 22:44:30.767488 13321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:44:30.762852 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45842 (local address 127.10.198.3:41889)
0925 22:44:30.763181 (+ 329us) server_negotiation.cc:206] Beginning negotiation
0925 22:44:30.763192 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 22:44:30.763208 (+ 16us) server_negotiation.cc:407] Connection header received
0925 22:44:30.763426 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:44:30.763429 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:44:30.763503 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:44:30.763662 (+ 159us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 22:44:30.763965 (+ 303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:44:30.764576 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:44:30.765491 (+ 915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:44:30.765739 (+ 248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:44:30.765864 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:44:30.766439 (+ 575us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 22:44:30.766970 (+ 531us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 22:44:30.767108 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0925 22:44:30.767175 (+ 67us) server_negotiation.cc:299] Negotiation successful
0925 22:44:30.767264 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":243,"thread_start_us":133,"threads_started":1}
I20260925 22:44:30.780256 13264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:44:30.775801 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.198.3:51853 (local address 127.10.198.2:45425)
0925 22:44:30.775976 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0925 22:44:30.775980 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:44:30.776270 (+ 290us) server_negotiation.cc:407] Connection header received
0925 22:44:30.776436 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:44:30.776439 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:44:30.776496 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:44:30.776586 (+ 90us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 22:44:30.777343 (+ 757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:44:30.777850 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:44:30.778805 (+ 955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:44:30.779043 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:44:30.779160 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:44:30.779510 (+ 350us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 22:44:30.779594 (+ 84us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 22:44:30.779678 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0925 22:44:30.780008 (+ 330us) server_negotiation.cc:299] Negotiation successful
0925 22:44:30.780070 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260925 22:44:30.780306 13324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:44:30.775701 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.198.2:45425 (local address 127.10.198.3:51853)
0925 22:44:30.776143 (+ 442us) negotiation.cc:107] Waiting for socket to connect
0925 22:44:30.776157 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 22:44:30.776318 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:44:30.776600 (+ 282us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:44:30.776602 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:44:30.776611 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 22:44:30.777106 (+ 495us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:44:30.777113 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:44:30.777992 (+ 879us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:44:30.777995 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:44:30.778646 (+ 651us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:44:30.778655 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:44:30.779171 (+ 516us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:44:30.779175 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:44:30.779273 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:44:30.779327 (+ 54us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 22:44:30.779687 (+ 360us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 22:44:30.779696 (+ 9us) client_negotiation.cc:769] Sending connection context
0925 22:44:30.779992 (+ 296us) client_negotiation.cc:240] Negotiation successful
0925 22:44:30.780100 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":337,"thread_start_us":146,"threads_started":1}
I20260925 22:44:30.788483 13060 raft_consensus.cc:1261] T aa29a660ca9044f3b387a685651cdc7f P 2a585751d44c49f98359ac6423e39c71 [term 1 FOLLOWER]: Refusing update from remote peer c1b2193f640945a895e6497c6422849f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 22:44:30.788755 12919 raft_consensus.cc:1261] T aa29a660ca9044f3b387a685651cdc7f P 3cad46077ef143feae0d7ee5ac6852f4 [term 1 FOLLOWER]: Refusing update from remote peer c1b2193f640945a895e6497c6422849f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 22:44:30.789311 13319 consensus_queue.cc:1035] T aa29a660ca9044f3b387a685651cdc7f P c1b2193f640945a895e6497c6422849f [LEADER]: Connected to new peer: Peer: permanent_uuid: "3cad46077ef143feae0d7ee5ac6852f4" member_type: VOTER last_known_addr { host: "127.10.198.1" port: 36901 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 22:44:30.789520 13277 consensus_queue.cc:1035] T aa29a660ca9044f3b387a685651cdc7f P c1b2193f640945a895e6497c6422849f [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a585751d44c49f98359ac6423e39c71" member_type: VOTER last_known_addr { host: "127.10.198.2" port: 45425 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 22:44:30.812983 13306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:44:30.808965 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.198.3:41889 (local address 127.10.198.2:33155)
0925 22:44:30.809227 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0925 22:44:30.809243 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 22:44:30.809330 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:44:30.809699 (+ 369us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:44:30.809704 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:44:30.809713 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 22:44:30.809819 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:44:30.809825 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:44:30.810667 (+ 842us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:44:30.810672 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:44:30.811287 (+ 615us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:44:30.811296 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:44:30.811773 (+ 477us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:44:30.811777 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:44:30.811900 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:44:30.811958 (+ 58us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 22:44:30.812626 (+ 668us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 22:44:30.812633 (+ 7us) client_negotiation.cc:769] Sending connection context
0925 22:44:30.812702 (+ 69us) client_negotiation.cc:240] Negotiation successful
0925 22:44:30.812795 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":158}
I20260925 22:44:30.813529 13321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:44:30.809164 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.198.2:33155 (local address 127.10.198.3:41889)
0925 22:44:30.809285 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0925 22:44:30.809290 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:44:30.809309 (+ 19us) server_negotiation.cc:407] Connection header received
0925 22:44:30.809438 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:44:30.809440 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:44:30.809480 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:44:30.809577 (+ 97us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 22:44:30.809953 (+ 376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:44:30.810554 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:44:30.811422 (+ 868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:44:30.811654 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:44:30.811770 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:44:30.812099 (+ 329us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 22:44:30.812209 (+ 110us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 22:44:30.812292 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0925 22:44:30.813324 (+ 1032us) server_negotiation.cc:299] Negotiation successful
0925 22:44:30.813391 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
W20260925 22:44:30.814195 13295 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260925 22:44:32.389475 12762 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42286:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 22:44:32.389676 12762 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42286:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 22:44:32.392815 12762 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9f0e816c575a42caac240acda127e507: Sending DeleteTablet for 3 replicas of tablet aa29a660ca9044f3b387a685651cdc7f
I20260925 22:44:32.393559 12899 tablet_service.cc:1514] Processing DeleteTablet for tablet aa29a660ca9044f3b387a685651cdc7f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 22:44:32 UTC) from {username='kudu', principal='kudu/127.10.198.62@KRBTEST.COM'} at 127.0.0.1:38708
I20260925 22:44:32.393663 13040 tablet_service.cc:1514] Processing DeleteTablet for tablet aa29a660ca9044f3b387a685651cdc7f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 22:44:32 UTC) from {username='kudu', principal='kudu/127.10.198.62@KRBTEST.COM'} at 127.0.0.1:33700
I20260925 22:44:32.393695 13180 tablet_service.cc:1514] Processing DeleteTablet for tablet aa29a660ca9044f3b387a685651cdc7f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 22:44:32 UTC) from {username='kudu', principal='kudu/127.10.198.62@KRBTEST.COM'} at 127.0.0.1:45840
I20260925 22:44:32.394014 13352 tablet_replica.cc:330] T aa29a660ca9044f3b387a685651cdc7f P 3cad46077ef143feae0d7ee5ac6852f4: stopping tablet replica
I20260925 22:44:32.394127 13354 tablet_replica.cc:330] T aa29a660ca9044f3b387a685651cdc7f P c1b2193f640945a895e6497c6422849f: stopping tablet replica
I20260925 22:44:32.394176 13352 raft_consensus.cc:2229] T aa29a660ca9044f3b387a685651cdc7f P 3cad46077ef143feae0d7ee5ac6852f4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 22:44:32.394274 13354 raft_consensus.cc:2229] T aa29a660ca9044f3b387a685651cdc7f P c1b2193f640945a895e6497c6422849f [term 1 LEADER]: Raft consensus shutting down.
I20260925 22:44:32.394313 13352 raft_consensus.cc:2258] T aa29a660ca9044f3b387a685651cdc7f P 3cad46077ef143feae0d7ee5ac6852f4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 22:44:32.394531 13353 tablet_replica.cc:330] T aa29a660ca9044f3b387a685651cdc7f P 2a585751d44c49f98359ac6423e39c71: stopping tablet replica
I20260925 22:44:32.394709 13354 raft_consensus.cc:2258] T aa29a660ca9044f3b387a685651cdc7f P c1b2193f640945a895e6497c6422849f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 22:44:32.394716 13353 raft_consensus.cc:2229] T aa29a660ca9044f3b387a685651cdc7f P 2a585751d44c49f98359ac6423e39c71 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 22:44:32.395066 13353 raft_consensus.cc:2258] T aa29a660ca9044f3b387a685651cdc7f P 2a585751d44c49f98359ac6423e39c71 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 22:44:32.395120 13352 ts_tablet_manager.cc:1905] T aa29a660ca9044f3b387a685651cdc7f P 3cad46077ef143feae0d7ee5ac6852f4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 22:44:32.395632 13354 ts_tablet_manager.cc:1905] T aa29a660ca9044f3b387a685651cdc7f P c1b2193f640945a895e6497c6422849f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 22:44:32.397833 13353 ts_tablet_manager.cc:1905] T aa29a660ca9044f3b387a685651cdc7f P 2a585751d44c49f98359ac6423e39c71: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 22:44:32.398010 13352 ts_tablet_manager.cc:1918] T aa29a660ca9044f3b387a685651cdc7f P 3cad46077ef143feae0d7ee5ac6852f4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260925 22:44:32.398100 13352 log.cc:1196] T aa29a660ca9044f3b387a685651cdc7f P 3cad46077ef143feae0d7ee5ac6852f4: Deleting WAL directory at /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790376259650642-11032-0/minicluster-data/ts-0/wal/wals/aa29a660ca9044f3b387a685651cdc7f
I20260925 22:44:32.398414 13352 ts_tablet_manager.cc:1939] T aa29a660ca9044f3b387a685651cdc7f P 3cad46077ef143feae0d7ee5ac6852f4: Deleting consensus metadata
I20260925 22:44:32.398789 13354 ts_tablet_manager.cc:1918] T aa29a660ca9044f3b387a685651cdc7f P c1b2193f640945a895e6497c6422849f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260925 22:44:32.398880 13354 log.cc:1196] T aa29a660ca9044f3b387a685651cdc7f P c1b2193f640945a895e6497c6422849f: Deleting WAL directory at /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790376259650642-11032-0/minicluster-data/ts-2/wal/wals/aa29a660ca9044f3b387a685651cdc7f
I20260925 22:44:32.399204 13354 ts_tablet_manager.cc:1939] T aa29a660ca9044f3b387a685651cdc7f P c1b2193f640945a895e6497c6422849f: Deleting consensus metadata
I20260925 22:44:32.399345 12749 catalog_manager.cc:4841] TS 3cad46077ef143feae0d7ee5ac6852f4 (127.10.198.1:36901): tablet aa29a660ca9044f3b387a685651cdc7f (table test-table [id=f3a861f0125a4777be949439e8b0cba5]) successfully deleted
I20260925 22:44:32.400092 12747 catalog_manager.cc:4841] TS c1b2193f640945a895e6497c6422849f (127.10.198.3:41889): tablet aa29a660ca9044f3b387a685651cdc7f (table test-table [id=f3a861f0125a4777be949439e8b0cba5]) successfully deleted
I20260925 22:44:32.400581 13353 ts_tablet_manager.cc:1918] T aa29a660ca9044f3b387a685651cdc7f P 2a585751d44c49f98359ac6423e39c71: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260925 22:44:32.400653 13353 log.cc:1196] T aa29a660ca9044f3b387a685651cdc7f P 2a585751d44c49f98359ac6423e39c71: Deleting WAL directory at /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790376259650642-11032-0/minicluster-data/ts-1/wal/wals/aa29a660ca9044f3b387a685651cdc7f
I20260925 22:44:32.401376 13353 ts_tablet_manager.cc:1939] T aa29a660ca9044f3b387a685651cdc7f P 2a585751d44c49f98359ac6423e39c71: Deleting consensus metadata
I20260925 22:44:32.402271 12748 catalog_manager.cc:4841] TS 2a585751d44c49f98359ac6423e39c71 (127.10.198.2:45425): tablet aa29a660ca9044f3b387a685651cdc7f (table test-table [id=f3a861f0125a4777be949439e8b0cba5]) successfully deleted
Sep 25 22:44:32 dist-test-slave-5vdr krb5kdc[12718](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376272, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 25 22:44:32 dist-test-slave-5vdr krb5kdc[12718](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376272, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.10.198.62@KRBTEST.COM
I20260925 22:44:32.430727 13362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:44:32.416223 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36962 (local address 127.10.198.62:42121)
0925 22:44:32.416686 (+ 463us) server_negotiation.cc:206] Beginning negotiation
0925 22:44:32.416702 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0925 22:44:32.416723 (+ 21us) server_negotiation.cc:407] Connection header received
0925 22:44:32.416782 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:44:32.416785 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:44:32.416842 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:44:32.416930 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:44:32.417827 (+ 897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:44:32.418417 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:44:32.419322 (+ 905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:44:32.419584 (+ 262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:44:32.419677 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:44:32.426792 (+ 7115us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:44:32.426796 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:44:32.426799 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:44:32.426836 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:44:32.428637 (+ 1801us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:44:32.429267 (+ 630us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:44:32.429271 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:44:32.429272 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:44:32.429376 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:44:32.429704 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:44:32.429707 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:44:32.429709 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:44:32.429929 (+ 220us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:44:32.430044 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0925 22:44:32.430347 (+ 303us) server_negotiation.cc:299] Negotiation successful
0925 22:44:32.430472 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":254,"thread_start_us":138,"threads_started":1}
I20260925 22:44:32.434010 12762 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:36962:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 22:44:32.434530 12762 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 22:44:32.439818 13040 tablet_service.cc:1467] Processing CreateTablet for tablet 97edf07635ca428abc52478177b12273 (DEFAULT_TABLE table=test-table [id=ed2cf039e86e4da480c03b9f8e69f494]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 22:44:32.439824 12899 tablet_service.cc:1467] Processing CreateTablet for tablet 97edf07635ca428abc52478177b12273 (DEFAULT_TABLE table=test-table [id=ed2cf039e86e4da480c03b9f8e69f494]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 22:44:32.440214 12899 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 97edf07635ca428abc52478177b12273. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:44:32.440168 13180 tablet_service.cc:1467] Processing CreateTablet for tablet 97edf07635ca428abc52478177b12273 (DEFAULT_TABLE table=test-table [id=ed2cf039e86e4da480c03b9f8e69f494]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 22:44:32.440231 13040 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 97edf07635ca428abc52478177b12273. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:44:32.440420 13180 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 97edf07635ca428abc52478177b12273. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:44:32.443533 13363 tablet_bootstrap.cc:492] T 97edf07635ca428abc52478177b12273 P 2a585751d44c49f98359ac6423e39c71: Bootstrap starting.
I20260925 22:44:32.443724 13364 tablet_bootstrap.cc:492] T 97edf07635ca428abc52478177b12273 P 3cad46077ef143feae0d7ee5ac6852f4: Bootstrap starting.
I20260925 22:44:32.444149 13365 tablet_bootstrap.cc:492] T 97edf07635ca428abc52478177b12273 P c1b2193f640945a895e6497c6422849f: Bootstrap starting.
I20260925 22:44:32.444700 13363 tablet_bootstrap.cc:654] T 97edf07635ca428abc52478177b12273 P 2a585751d44c49f98359ac6423e39c71: Neither blocks nor log segments found. Creating new log.
I20260925 22:44:32.445032 13364 tablet_bootstrap.cc:654] T 97edf07635ca428abc52478177b12273 P 3cad46077ef143feae0d7ee5ac6852f4: Neither blocks nor log segments found. Creating new log.
I20260925 22:44:32.445103 13365 tablet_bootstrap.cc:654] T 97edf07635ca428abc52478177b12273 P c1b2193f640945a895e6497c6422849f: Neither blocks nor log segments found. Creating new log.
I20260925 22:44:32.446357 13365 tablet_bootstrap.cc:492] T 97edf07635ca428abc52478177b12273 P c1b2193f640945a895e6497c6422849f: No bootstrap required, opened a new log
I20260925 22:44:32.446386 13363 tablet_bootstrap.cc:492] T 97edf07635ca428abc52478177b12273 P 2a585751d44c49f98359ac6423e39c71: No bootstrap required, opened a new log
I20260925 22:44:32.446484 13363 ts_tablet_manager.cc:1397] T 97edf07635ca428abc52478177b12273 P 2a585751d44c49f98359ac6423e39c71: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260925 22:44:32.446484 13365 ts_tablet_manager.cc:1397] T 97edf07635ca428abc52478177b12273 P c1b2193f640945a895e6497c6422849f: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260925 22:44:32.446954 13365 raft_consensus.cc:348] T 97edf07635ca428abc52478177b12273 P c1b2193f640945a895e6497c6422849f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cad46077ef143feae0d7ee5ac6852f4" member_type: VOTER last_known_addr { host: "127.10.198.1" port: 36901 } } peers { permanent_uuid: "2a585751d44c49f98359ac6423e39c71" member_type: VOTER last_known_addr { host: "127.10.198.2" port: 45425 } } peers { permanent_uuid: "c1b2193f640945a895e6497c6422849f" member_type: VOTER last_known_addr { host: "127.10.198.3" port: 41889 } }
I20260925 22:44:32.446965 13363 raft_consensus.cc:348] T 97edf07635ca428abc52478177b12273 P 2a585751d44c49f98359ac6423e39c71 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cad46077ef143feae0d7ee5ac6852f4" member_type: VOTER last_known_addr { host: "127.10.198.1" port: 36901 } } peers { permanent_uuid: "2a585751d44c49f98359ac6423e39c71" member_type: VOTER last_known_addr { host: "127.10.198.2" port: 45425 } } peers { permanent_uuid: "c1b2193f640945a895e6497c6422849f" member_type: VOTER last_known_addr { host: "127.10.198.3" port: 41889 } }
I20260925 22:44:32.447089 13365 raft_consensus.cc:374] T 97edf07635ca428abc52478177b12273 P c1b2193f640945a895e6497c6422849f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:44:32.447086 13363 raft_consensus.cc:374] T 97edf07635ca428abc52478177b12273 P 2a585751d44c49f98359ac6423e39c71 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:44:32.447135 13363 raft_consensus.cc:729] T 97edf07635ca428abc52478177b12273 P 2a585751d44c49f98359ac6423e39c71 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2a585751d44c49f98359ac6423e39c71, State: Initialized, Role: FOLLOWER
I20260925 22:44:32.447135 13365 raft_consensus.cc:729] T 97edf07635ca428abc52478177b12273 P c1b2193f640945a895e6497c6422849f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c1b2193f640945a895e6497c6422849f, State: Initialized, Role: FOLLOWER
I20260925 22:44:32.447276 13363 consensus_queue.cc:260] T 97edf07635ca428abc52478177b12273 P 2a585751d44c49f98359ac6423e39c71 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cad46077ef143feae0d7ee5ac6852f4" member_type: VOTER last_known_addr { host: "127.10.198.1" port: 36901 } } peers { permanent_uuid: "2a585751d44c49f98359ac6423e39c71" member_type: VOTER last_known_addr { host: "127.10.198.2" port: 45425 } } peers { permanent_uuid: "c1b2193f640945a895e6497c6422849f" member_type: VOTER last_known_addr { host: "127.10.198.3" port: 41889 } }
I20260925 22:44:32.447321 13365 consensus_queue.cc:260] T 97edf07635ca428abc52478177b12273 P c1b2193f640945a895e6497c6422849f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cad46077ef143feae0d7ee5ac6852f4" member_type: VOTER last_known_addr { host: "127.10.198.1" port: 36901 } } peers { permanent_uuid: "2a585751d44c49f98359ac6423e39c71" member_type: VOTER last_known_addr { host: "127.10.198.2" port: 45425 } } peers { permanent_uuid: "c1b2193f640945a895e6497c6422849f" member_type: VOTER last_known_addr { host: "127.10.198.3" port: 41889 } }
I20260925 22:44:32.447570 13363 ts_tablet_manager.cc:1428] T 97edf07635ca428abc52478177b12273 P 2a585751d44c49f98359ac6423e39c71: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260925 22:44:32.447713 13365 ts_tablet_manager.cc:1428] T 97edf07635ca428abc52478177b12273 P c1b2193f640945a895e6497c6422849f: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260925 22:44:32.448412 13364 tablet_bootstrap.cc:492] T 97edf07635ca428abc52478177b12273 P 3cad46077ef143feae0d7ee5ac6852f4: No bootstrap required, opened a new log
I20260925 22:44:32.448520 13364 ts_tablet_manager.cc:1397] T 97edf07635ca428abc52478177b12273 P 3cad46077ef143feae0d7ee5ac6852f4: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20260925 22:44:32.449064 13364 raft_consensus.cc:348] T 97edf07635ca428abc52478177b12273 P 3cad46077ef143feae0d7ee5ac6852f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cad46077ef143feae0d7ee5ac6852f4" member_type: VOTER last_known_addr { host: "127.10.198.1" port: 36901 } } peers { permanent_uuid: "2a585751d44c49f98359ac6423e39c71" member_type: VOTER last_known_addr { host: "127.10.198.2" port: 45425 } } peers { permanent_uuid: "c1b2193f640945a895e6497c6422849f" member_type: VOTER last_known_addr { host: "127.10.198.3" port: 41889 } }
I20260925 22:44:32.449199 13364 raft_consensus.cc:374] T 97edf07635ca428abc52478177b12273 P 3cad46077ef143feae0d7ee5ac6852f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:44:32.449249 13364 raft_consensus.cc:729] T 97edf07635ca428abc52478177b12273 P 3cad46077ef143feae0d7ee5ac6852f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3cad46077ef143feae0d7ee5ac6852f4, State: Initialized, Role: FOLLOWER
I20260925 22:44:32.449388 13364 consensus_queue.cc:260] T 97edf07635ca428abc52478177b12273 P 3cad46077ef143feae0d7ee5ac6852f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cad46077ef143feae0d7ee5ac6852f4" member_type: VOTER last_known_addr { host: "127.10.198.1" port: 36901 } } peers { permanent_uuid: "2a585751d44c49f98359ac6423e39c71" member_type: VOTER last_known_addr { host: "127.10.198.2" port: 45425 } } peers { permanent_uuid: "c1b2193f640945a895e6497c6422849f" member_type: VOTER last_known_addr { host: "127.10.198.3" port: 41889 } }
I20260925 22:44:32.449831 13364 ts_tablet_manager.cc:1428] T 97edf07635ca428abc52478177b12273 P 3cad46077ef143feae0d7ee5ac6852f4: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20260925 22:44:32.575965 13344 raft_consensus.cc:482] T 97edf07635ca428abc52478177b12273 P c1b2193f640945a895e6497c6422849f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 22:44:32.576189 13344 raft_consensus.cc:504] T 97edf07635ca428abc52478177b12273 P c1b2193f640945a895e6497c6422849f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cad46077ef143feae0d7ee5ac6852f4" member_type: VOTER last_known_addr { host: "127.10.198.1" port: 36901 } } peers { permanent_uuid: "2a585751d44c49f98359ac6423e39c71" member_type: VOTER last_known_addr { host: "127.10.198.2" port: 45425 } } peers { permanent_uuid: "c1b2193f640945a895e6497c6422849f" member_type: VOTER last_known_addr { host: "127.10.198.3" port: 41889 } }
I20260925 22:44:32.576713 13344 leader_election.cc:288] T 97edf07635ca428abc52478177b12273 P c1b2193f640945a895e6497c6422849f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3cad46077ef143feae0d7ee5ac6852f4 (127.10.198.1:36901), 2a585751d44c49f98359ac6423e39c71 (127.10.198.2:45425)
I20260925 22:44:32.577275 13060 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "97edf07635ca428abc52478177b12273" candidate_uuid: "c1b2193f640945a895e6497c6422849f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a585751d44c49f98359ac6423e39c71" is_pre_election: true
I20260925 22:44:32.577273 12919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "97edf07635ca428abc52478177b12273" candidate_uuid: "c1b2193f640945a895e6497c6422849f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3cad46077ef143feae0d7ee5ac6852f4" is_pre_election: true
I20260925 22:44:32.577462 12919 raft_consensus.cc:2454] T 97edf07635ca428abc52478177b12273 P 3cad46077ef143feae0d7ee5ac6852f4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c1b2193f640945a895e6497c6422849f in term 0.
I20260925 22:44:32.577457 13060 raft_consensus.cc:2454] T 97edf07635ca428abc52478177b12273 P 2a585751d44c49f98359ac6423e39c71 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c1b2193f640945a895e6497c6422849f in term 0.
I20260925 22:44:32.577977 13133 leader_election.cc:302] T 97edf07635ca428abc52478177b12273 P c1b2193f640945a895e6497c6422849f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 2a585751d44c49f98359ac6423e39c71, 3cad46077ef143feae0d7ee5ac6852f4, c1b2193f640945a895e6497c6422849f; no voters:
I20260925 22:44:32.578202 13344 raft_consensus.cc:2781] T 97edf07635ca428abc52478177b12273 P c1b2193f640945a895e6497c6422849f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 22:44:32.578271 13344 raft_consensus.cc:482] T 97edf07635ca428abc52478177b12273 P c1b2193f640945a895e6497c6422849f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 22:44:32.578299 13344 raft_consensus.cc:3037] T 97edf07635ca428abc52478177b12273 P c1b2193f640945a895e6497c6422849f [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:44:32.579370 13344 raft_consensus.cc:504] T 97edf07635ca428abc52478177b12273 P c1b2193f640945a895e6497c6422849f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cad46077ef143feae0d7ee5ac6852f4" member_type: VOTER last_known_addr { host: "127.10.198.1" port: 36901 } } peers { permanent_uuid: "2a585751d44c49f98359ac6423e39c71" member_type: VOTER last_known_addr { host: "127.10.198.2" port: 45425 } } peers { permanent_uuid: "c1b2193f640945a895e6497c6422849f" member_type: VOTER last_known_addr { host: "127.10.198.3" port: 41889 } }
I20260925 22:44:32.579775 13344 leader_election.cc:288] T 97edf07635ca428abc52478177b12273 P c1b2193f640945a895e6497c6422849f [CANDIDATE]: Term 1 election: Requested vote from peers 3cad46077ef143feae0d7ee5ac6852f4 (127.10.198.1:36901), 2a585751d44c49f98359ac6423e39c71 (127.10.198.2:45425)
I20260925 22:44:32.580238 13060 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "97edf07635ca428abc52478177b12273" candidate_uuid: "c1b2193f640945a895e6497c6422849f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a585751d44c49f98359ac6423e39c71"
I20260925 22:44:32.580240 12919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "97edf07635ca428abc52478177b12273" candidate_uuid: "c1b2193f640945a895e6497c6422849f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3cad46077ef143feae0d7ee5ac6852f4"
I20260925 22:44:32.580367 13060 raft_consensus.cc:3037] T 97edf07635ca428abc52478177b12273 P 2a585751d44c49f98359ac6423e39c71 [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:44:32.580394 12919 raft_consensus.cc:3037] T 97edf07635ca428abc52478177b12273 P 3cad46077ef143feae0d7ee5ac6852f4 [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:44:32.581384 12919 raft_consensus.cc:2454] T 97edf07635ca428abc52478177b12273 P 3cad46077ef143feae0d7ee5ac6852f4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c1b2193f640945a895e6497c6422849f in term 1.
I20260925 22:44:32.581388 13060 raft_consensus.cc:2454] T 97edf07635ca428abc52478177b12273 P 2a585751d44c49f98359ac6423e39c71 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c1b2193f640945a895e6497c6422849f in term 1.
I20260925 22:44:32.581769 13133 leader_election.cc:302] T 97edf07635ca428abc52478177b12273 P c1b2193f640945a895e6497c6422849f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3cad46077ef143feae0d7ee5ac6852f4, c1b2193f640945a895e6497c6422849f; no voters:
I20260925 22:44:32.582032 13344 raft_consensus.cc:2781] T 97edf07635ca428abc52478177b12273 P c1b2193f640945a895e6497c6422849f [term 1 FOLLOWER]: Leader election won for term 1
I20260925 22:44:32.582170 13344 raft_consensus.cc:686] T 97edf07635ca428abc52478177b12273 P c1b2193f640945a895e6497c6422849f [term 1 LEADER]: Becoming Leader. State: Replica: c1b2193f640945a895e6497c6422849f, State: Running, Role: LEADER
I20260925 22:44:32.582372 13344 consensus_queue.cc:237] T 97edf07635ca428abc52478177b12273 P c1b2193f640945a895e6497c6422849f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cad46077ef143feae0d7ee5ac6852f4" member_type: VOTER last_known_addr { host: "127.10.198.1" port: 36901 } } peers { permanent_uuid: "2a585751d44c49f98359ac6423e39c71" member_type: VOTER last_known_addr { host: "127.10.198.2" port: 45425 } } peers { permanent_uuid: "c1b2193f640945a895e6497c6422849f" member_type: VOTER last_known_addr { host: "127.10.198.3" port: 41889 } }
I20260925 22:44:32.584640 12763 catalog_manager.cc:5495] T 97edf07635ca428abc52478177b12273 P c1b2193f640945a895e6497c6422849f reported cstate change: term changed from 0 to 1, leader changed from <none> to c1b2193f640945a895e6497c6422849f (127.10.198.3). New cstate: current_term: 1 leader_uuid: "c1b2193f640945a895e6497c6422849f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cad46077ef143feae0d7ee5ac6852f4" member_type: VOTER last_known_addr { host: "127.10.198.1" port: 36901 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2a585751d44c49f98359ac6423e39c71" member_type: VOTER last_known_addr { host: "127.10.198.2" port: 45425 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c1b2193f640945a895e6497c6422849f" member_type: VOTER last_known_addr { host: "127.10.198.3" port: 41889 } health_report { overall_health: HEALTHY } } }
I20260925 22:44:32.616008 13372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:44:32.611353 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39214 (local address 127.10.198.3:41889)
0925 22:44:32.611676 (+ 323us) server_negotiation.cc:206] Beginning negotiation
0925 22:44:32.611685 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 22:44:32.611699 (+ 14us) server_negotiation.cc:407] Connection header received
0925 22:44:32.611747 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:44:32.611749 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:44:32.611794 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:44:32.611915 (+ 121us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 22:44:32.612357 (+ 442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:44:32.613176 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:44:32.614689 (+ 1513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:44:32.614976 (+ 287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:44:32.615082 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:44:32.615442 (+ 360us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 22:44:32.615574 (+ 132us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 22:44:32.615639 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0925 22:44:32.615791 (+ 152us) server_negotiation.cc:299] Negotiation successful
0925 22:44:32.615870 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":238,"thread_start_us":158,"threads_started":1}
I20260925 22:44:32.624413 12919 raft_consensus.cc:1261] T 97edf07635ca428abc52478177b12273 P 3cad46077ef143feae0d7ee5ac6852f4 [term 1 FOLLOWER]: Refusing update from remote peer c1b2193f640945a895e6497c6422849f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 22:44:32.624413 13060 raft_consensus.cc:1261] T 97edf07635ca428abc52478177b12273 P 2a585751d44c49f98359ac6423e39c71 [term 1 FOLLOWER]: Refusing update from remote peer c1b2193f640945a895e6497c6422849f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 22:44:32.625192 13344 consensus_queue.cc:1035] T 97edf07635ca428abc52478177b12273 P c1b2193f640945a895e6497c6422849f [LEADER]: Connected to new peer: Peer: permanent_uuid: "3cad46077ef143feae0d7ee5ac6852f4" member_type: VOTER last_known_addr { host: "127.10.198.1" port: 36901 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 22:44:32.625473 13341 consensus_queue.cc:1035] T 97edf07635ca428abc52478177b12273 P c1b2193f640945a895e6497c6422849f [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a585751d44c49f98359ac6423e39c71" member_type: VOTER last_known_addr { host: "127.10.198.2" port: 45425 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 22:44:32.664623 12763 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:36962:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 22:44:32.664791 12763 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:36962:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 22:44:32.667164 12763 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9f0e816c575a42caac240acda127e507: Sending DeleteTablet for 3 replicas of tablet 97edf07635ca428abc52478177b12273
I20260925 22:44:32.667987 12899 tablet_service.cc:1514] Processing DeleteTablet for tablet 97edf07635ca428abc52478177b12273 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 22:44:32 UTC) from {username='kudu', principal='kudu/127.10.198.62@KRBTEST.COM'} at 127.0.0.1:38708
I20260925 22:44:32.668239 13352 tablet_replica.cc:330] T 97edf07635ca428abc52478177b12273 P 3cad46077ef143feae0d7ee5ac6852f4: stopping tablet replica
I20260925 22:44:32.668385 13352 raft_consensus.cc:2229] T 97edf07635ca428abc52478177b12273 P 3cad46077ef143feae0d7ee5ac6852f4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 22:44:32.668532 13352 raft_consensus.cc:2258] T 97edf07635ca428abc52478177b12273 P 3cad46077ef143feae0d7ee5ac6852f4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 22:44:32.668753 13180 tablet_service.cc:1514] Processing DeleteTablet for tablet 97edf07635ca428abc52478177b12273 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 22:44:32 UTC) from {username='kudu', principal='kudu/127.10.198.62@KRBTEST.COM'} at 127.0.0.1:45840
I20260925 22:44:32.669031 13354 tablet_replica.cc:330] T 97edf07635ca428abc52478177b12273 P c1b2193f640945a895e6497c6422849f: stopping tablet replica
I20260925 22:44:32.669032 13040 tablet_service.cc:1514] Processing DeleteTablet for tablet 97edf07635ca428abc52478177b12273 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 22:44:32 UTC) from {username='kudu', principal='kudu/127.10.198.62@KRBTEST.COM'} at 127.0.0.1:33700
I20260925 22:44:32.669181 13354 raft_consensus.cc:2229] T 97edf07635ca428abc52478177b12273 P c1b2193f640945a895e6497c6422849f [term 1 LEADER]: Raft consensus shutting down.
I20260925 22:44:32.669243 13353 tablet_replica.cc:330] T 97edf07635ca428abc52478177b12273 P 2a585751d44c49f98359ac6423e39c71: stopping tablet replica
I20260925 22:44:32.669247 12763 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:36962:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260925 22:44:32.669407 13354 raft_consensus.cc:2258] T 97edf07635ca428abc52478177b12273 P c1b2193f640945a895e6497c6422849f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 22:44:32.669379 13353 raft_consensus.cc:2229] T 97edf07635ca428abc52478177b12273 P 2a585751d44c49f98359ac6423e39c71 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 22:44:32.669554 13353 raft_consensus.cc:2258] T 97edf07635ca428abc52478177b12273 P 2a585751d44c49f98359ac6423e39c71 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260925 22:44:32.669747 12763 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 22:44:32.670233 13354 ts_tablet_manager.cc:1905] T 97edf07635ca428abc52478177b12273 P c1b2193f640945a895e6497c6422849f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 22:44:32.670406 13353 ts_tablet_manager.cc:1905] T 97edf07635ca428abc52478177b12273 P 2a585751d44c49f98359ac6423e39c71: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 22:44:32.673077 13354 ts_tablet_manager.cc:1918] T 97edf07635ca428abc52478177b12273 P c1b2193f640945a895e6497c6422849f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 22:44:32.673168 13354 log.cc:1196] T 97edf07635ca428abc52478177b12273 P c1b2193f640945a895e6497c6422849f: Deleting WAL directory at /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790376259650642-11032-0/minicluster-data/ts-2/wal/wals/97edf07635ca428abc52478177b12273
I20260925 22:44:32.673417 13354 ts_tablet_manager.cc:1939] T 97edf07635ca428abc52478177b12273 P c1b2193f640945a895e6497c6422849f: Deleting consensus metadata
I20260925 22:44:32.673415 13353 ts_tablet_manager.cc:1918] T 97edf07635ca428abc52478177b12273 P 2a585751d44c49f98359ac6423e39c71: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 22:44:32.673511 13353 log.cc:1196] T 97edf07635ca428abc52478177b12273 P 2a585751d44c49f98359ac6423e39c71: Deleting WAL directory at /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790376259650642-11032-0/minicluster-data/ts-1/wal/wals/97edf07635ca428abc52478177b12273
I20260925 22:44:32.673785 13353 ts_tablet_manager.cc:1939] T 97edf07635ca428abc52478177b12273 P 2a585751d44c49f98359ac6423e39c71: Deleting consensus metadata
I20260925 22:44:32.674119 12747 catalog_manager.cc:4841] TS c1b2193f640945a895e6497c6422849f (127.10.198.3:41889): tablet 97edf07635ca428abc52478177b12273 (table test-table [id=ed2cf039e86e4da480c03b9f8e69f494]) successfully deleted
I20260925 22:44:32.674531 12748 catalog_manager.cc:4841] TS 2a585751d44c49f98359ac6423e39c71 (127.10.198.2:45425): tablet 97edf07635ca428abc52478177b12273 (table test-table [id=ed2cf039e86e4da480c03b9f8e69f494]) successfully deleted
I20260925 22:44:32.675993 13352 ts_tablet_manager.cc:1905] T 97edf07635ca428abc52478177b12273 P 3cad46077ef143feae0d7ee5ac6852f4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 22:44:32.677562 12899 tablet_service.cc:1467] Processing CreateTablet for tablet 6cfaf161dbe7482baadac04f70a5453d (DEFAULT_TABLE table=test-table [id=650c610e79a4447f80f75940f545196e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 22:44:32.677873 12899 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6cfaf161dbe7482baadac04f70a5453d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:44:32.677973 13040 tablet_service.cc:1467] Processing CreateTablet for tablet 6cfaf161dbe7482baadac04f70a5453d (DEFAULT_TABLE table=test-table [id=650c610e79a4447f80f75940f545196e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 22:44:32.678247 13040 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6cfaf161dbe7482baadac04f70a5453d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:44:32.678930 13352 ts_tablet_manager.cc:1918] T 97edf07635ca428abc52478177b12273 P 3cad46077ef143feae0d7ee5ac6852f4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 22:44:32.679018 13352 log.cc:1196] T 97edf07635ca428abc52478177b12273 P 3cad46077ef143feae0d7ee5ac6852f4: Deleting WAL directory at /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790376259650642-11032-0/minicluster-data/ts-0/wal/wals/97edf07635ca428abc52478177b12273
I20260925 22:44:32.679297 13352 ts_tablet_manager.cc:1939] T 97edf07635ca428abc52478177b12273 P 3cad46077ef143feae0d7ee5ac6852f4: Deleting consensus metadata
I20260925 22:44:32.679714 13364 tablet_bootstrap.cc:492] T 6cfaf161dbe7482baadac04f70a5453d P 3cad46077ef143feae0d7ee5ac6852f4: Bootstrap starting.
I20260925 22:44:32.680182 13180 tablet_service.cc:1467] Processing CreateTablet for tablet 6cfaf161dbe7482baadac04f70a5453d (DEFAULT_TABLE table=test-table [id=650c610e79a4447f80f75940f545196e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 22:44:32.680497 13180 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6cfaf161dbe7482baadac04f70a5453d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:44:32.680902 13364 tablet_bootstrap.cc:654] T 6cfaf161dbe7482baadac04f70a5453d P 3cad46077ef143feae0d7ee5ac6852f4: Neither blocks nor log segments found. Creating new log.
I20260925 22:44:32.681955 12749 catalog_manager.cc:4841] TS 3cad46077ef143feae0d7ee5ac6852f4 (127.10.198.1:36901): tablet 97edf07635ca428abc52478177b12273 (table test-table [id=ed2cf039e86e4da480c03b9f8e69f494]) successfully deleted
I20260925 22:44:32.682410 13364 tablet_bootstrap.cc:492] T 6cfaf161dbe7482baadac04f70a5453d P 3cad46077ef143feae0d7ee5ac6852f4: No bootstrap required, opened a new log
I20260925 22:44:32.682679 13364 ts_tablet_manager.cc:1397] T 6cfaf161dbe7482baadac04f70a5453d P 3cad46077ef143feae0d7ee5ac6852f4: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260925 22:44:32.683267 13364 raft_consensus.cc:348] T 6cfaf161dbe7482baadac04f70a5453d P 3cad46077ef143feae0d7ee5ac6852f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cad46077ef143feae0d7ee5ac6852f4" member_type: VOTER last_known_addr { host: "127.10.198.1" port: 36901 } } peers { permanent_uuid: "2a585751d44c49f98359ac6423e39c71" member_type: VOTER last_known_addr { host: "127.10.198.2" port: 45425 } } peers { permanent_uuid: "c1b2193f640945a895e6497c6422849f" member_type: VOTER last_known_addr { host: "127.10.198.3" port: 41889 } }
I20260925 22:44:32.683415 13364 raft_consensus.cc:374] T 6cfaf161dbe7482baadac04f70a5453d P 3cad46077ef143feae0d7ee5ac6852f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:44:32.683459 13364 raft_consensus.cc:729] T 6cfaf161dbe7482baadac04f70a5453d P 3cad46077ef143feae0d7ee5ac6852f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3cad46077ef143feae0d7ee5ac6852f4, State: Initialized, Role: FOLLOWER
I20260925 22:44:32.683465 13365 tablet_bootstrap.cc:492] T 6cfaf161dbe7482baadac04f70a5453d P c1b2193f640945a895e6497c6422849f: Bootstrap starting.
I20260925 22:44:32.683619 13364 consensus_queue.cc:260] T 6cfaf161dbe7482baadac04f70a5453d P 3cad46077ef143feae0d7ee5ac6852f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cad46077ef143feae0d7ee5ac6852f4" member_type: VOTER last_known_addr { host: "127.10.198.1" port: 36901 } } peers { permanent_uuid: "2a585751d44c49f98359ac6423e39c71" member_type: VOTER last_known_addr { host: "127.10.198.2" port: 45425 } } peers { permanent_uuid: "c1b2193f640945a895e6497c6422849f" member_type: VOTER last_known_addr { host: "127.10.198.3" port: 41889 } }
I20260925 22:44:32.683976 13364 ts_tablet_manager.cc:1428] T 6cfaf161dbe7482baadac04f70a5453d P 3cad46077ef143feae0d7ee5ac6852f4: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260925 22:44:32.684489 13365 tablet_bootstrap.cc:654] T 6cfaf161dbe7482baadac04f70a5453d P c1b2193f640945a895e6497c6422849f: Neither blocks nor log segments found. Creating new log.
I20260925 22:44:32.685441 13369 raft_consensus.cc:482] T 6cfaf161dbe7482baadac04f70a5453d P 3cad46077ef143feae0d7ee5ac6852f4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 22:44:32.685550 13369 raft_consensus.cc:504] T 6cfaf161dbe7482baadac04f70a5453d P 3cad46077ef143feae0d7ee5ac6852f4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cad46077ef143feae0d7ee5ac6852f4" member_type: VOTER last_known_addr { host: "127.10.198.1" port: 36901 } } peers { permanent_uuid: "2a585751d44c49f98359ac6423e39c71" member_type: VOTER last_known_addr { host: "127.10.198.2" port: 45425 } } peers { permanent_uuid: "c1b2193f640945a895e6497c6422849f" member_type: VOTER last_known_addr { host: "127.10.198.3" port: 41889 } }
I20260925 22:44:32.686265 13363 tablet_bootstrap.cc:492] T 6cfaf161dbe7482baadac04f70a5453d P 2a585751d44c49f98359ac6423e39c71: Bootstrap starting.
I20260925 22:44:32.686513 13365 tablet_bootstrap.cc:492] T 6cfaf161dbe7482baadac04f70a5453d P c1b2193f640945a895e6497c6422849f: No bootstrap required, opened a new log
I20260925 22:44:32.686637 13365 ts_tablet_manager.cc:1397] T 6cfaf161dbe7482baadac04f70a5453d P c1b2193f640945a895e6497c6422849f: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260925 22:44:32.687037 13365 raft_consensus.cc:348] T 6cfaf161dbe7482baadac04f70a5453d P c1b2193f640945a895e6497c6422849f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cad46077ef143feae0d7ee5ac6852f4" member_type: VOTER last_known_addr { host: "127.10.198.1" port: 36901 } } peers { permanent_uuid: "2a585751d44c49f98359ac6423e39c71" member_type: VOTER last_known_addr { host: "127.10.198.2" port: 45425 } } peers { permanent_uuid: "c1b2193f640945a895e6497c6422849f" member_type: VOTER last_known_addr { host: "127.10.198.3" port: 41889 } }
I20260925 22:44:32.687178 13365 raft_consensus.cc:374] T 6cfaf161dbe7482baadac04f70a5453d P c1b2193f640945a895e6497c6422849f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:44:32.687224 13365 raft_consensus.cc:729] T 6cfaf161dbe7482baadac04f70a5453d P c1b2193f640945a895e6497c6422849f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c1b2193f640945a895e6497c6422849f, State: Initialized, Role: FOLLOWER
I20260925 22:44:32.687251 13363 tablet_bootstrap.cc:654] T 6cfaf161dbe7482baadac04f70a5453d P 2a585751d44c49f98359ac6423e39c71: Neither blocks nor log segments found. Creating new log.
I20260925 22:44:32.687361 13365 consensus_queue.cc:260] T 6cfaf161dbe7482baadac04f70a5453d P c1b2193f640945a895e6497c6422849f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cad46077ef143feae0d7ee5ac6852f4" member_type: VOTER last_known_addr { host: "127.10.198.1" port: 36901 } } peers { permanent_uuid: "2a585751d44c49f98359ac6423e39c71" member_type: VOTER last_known_addr { host: "127.10.198.2" port: 45425 } } peers { permanent_uuid: "c1b2193f640945a895e6497c6422849f" member_type: VOTER last_known_addr { host: "127.10.198.3" port: 41889 } }
I20260925 22:44:32.687661 13365 ts_tablet_manager.cc:1428] T 6cfaf161dbe7482baadac04f70a5453d P c1b2193f640945a895e6497c6422849f: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260925 22:44:32.688697 13363 tablet_bootstrap.cc:492] T 6cfaf161dbe7482baadac04f70a5453d P 2a585751d44c49f98359ac6423e39c71: No bootstrap required, opened a new log
I20260925 22:44:32.688829 13363 ts_tablet_manager.cc:1397] T 6cfaf161dbe7482baadac04f70a5453d P 2a585751d44c49f98359ac6423e39c71: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260925 22:44:32.689366 13363 raft_consensus.cc:348] T 6cfaf161dbe7482baadac04f70a5453d P 2a585751d44c49f98359ac6423e39c71 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cad46077ef143feae0d7ee5ac6852f4" member_type: VOTER last_known_addr { host: "127.10.198.1" port: 36901 } } peers { permanent_uuid: "2a585751d44c49f98359ac6423e39c71" member_type: VOTER last_known_addr { host: "127.10.198.2" port: 45425 } } peers { permanent_uuid: "c1b2193f640945a895e6497c6422849f" member_type: VOTER last_known_addr { host: "127.10.198.3" port: 41889 } }
I20260925 22:44:32.689509 13363 raft_consensus.cc:374] T 6cfaf161dbe7482baadac04f70a5453d P 2a585751d44c49f98359ac6423e39c71 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:44:32.689551 13363 raft_consensus.cc:729] T 6cfaf161dbe7482baadac04f70a5453d P 2a585751d44c49f98359ac6423e39c71 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2a585751d44c49f98359ac6423e39c71, State: Initialized, Role: FOLLOWER
I20260925 22:44:32.689725 13363 consensus_queue.cc:260] T 6cfaf161dbe7482baadac04f70a5453d P 2a585751d44c49f98359ac6423e39c71 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cad46077ef143feae0d7ee5ac6852f4" member_type: VOTER last_known_addr { host: "127.10.198.1" port: 36901 } } peers { permanent_uuid: "2a585751d44c49f98359ac6423e39c71" member_type: VOTER last_known_addr { host: "127.10.198.2" port: 45425 } } peers { permanent_uuid: "c1b2193f640945a895e6497c6422849f" member_type: VOTER last_known_addr { host: "127.10.198.3" port: 41889 } }
I20260925 22:44:32.690166 13363 ts_tablet_manager.cc:1428] T 6cfaf161dbe7482baadac04f70a5453d P 2a585751d44c49f98359ac6423e39c71: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20260925 22:44:32.690892 13369 leader_election.cc:288] T 6cfaf161dbe7482baadac04f70a5453d P 3cad46077ef143feae0d7ee5ac6852f4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2a585751d44c49f98359ac6423e39c71 (127.10.198.2:45425), c1b2193f640945a895e6497c6422849f (127.10.198.3:41889)
I20260925 22:44:32.694798 13380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:44:32.689620 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.198.2:45425 (local address 127.10.198.1:36089)
0925 22:44:32.689931 (+ 311us) negotiation.cc:107] Waiting for socket to connect
0925 22:44:32.689949 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0925 22:44:32.690059 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:44:32.690906 (+ 847us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:44:32.690909 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:44:32.690932 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:44:32.691084 (+ 152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:44:32.691096 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:44:32.692240 (+ 1144us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:44:32.692246 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:44:32.693239 (+ 993us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:44:32.693248 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:44:32.694292 (+ 1044us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:44:32.694296 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:44:32.694430 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:44:32.694462 (+ 32us) client_negotiation.cc:769] Sending connection context
0925 22:44:32.694538 (+ 76us) client_negotiation.cc:240] Negotiation successful
0925 22:44:32.694624 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":154,"thread_start_us":86,"threads_started":1}
I20260925 22:44:32.695339 13379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:44:32.689392 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.198.1:36089 (local address 127.10.198.2:45425)
0925 22:44:32.690447 (+ 1055us) server_negotiation.cc:206] Beginning negotiation
0925 22:44:32.690466 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0925 22:44:32.690484 (+ 18us) server_negotiation.cc:407] Connection header received
0925 22:44:32.690538 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:44:32.690541 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:44:32.690590 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:44:32.690708 (+ 118us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:44:32.691245 (+ 537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:44:32.692074 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:44:32.693414 (+ 1340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:44:32.693992 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:44:32.694167 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:44:32.694227 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0925 22:44:32.695097 (+ 870us) server_negotiation.cc:299] Negotiation successful
0925 22:44:32.695198 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":958,"thread_start_us":90,"threads_started":1}
I20260925 22:44:32.695426 13381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:44:32.690113 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.198.3:41889 (local address 127.10.198.1:56061)
0925 22:44:32.691167 (+ 1054us) negotiation.cc:107] Waiting for socket to connect
0925 22:44:32.691182 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 22:44:32.691285 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:44:32.692004 (+ 719us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:44:32.692007 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:44:32.692025 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:44:32.692279 (+ 254us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:44:32.692288 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:44:32.693086 (+ 798us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:44:32.693089 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:44:32.694022 (+ 933us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:44:32.694031 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:44:32.694894 (+ 863us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:44:32.694898 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:44:32.695137 (+ 239us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:44:32.695161 (+ 24us) client_negotiation.cc:769] Sending connection context
0925 22:44:32.695223 (+ 62us) client_negotiation.cc:240] Negotiation successful
0925 22:44:32.695288 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":986,"thread_start_us":155,"threads_started":1}
I20260925 22:44:32.695678 13372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:44:32.690829 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.198.1:56061 (local address 127.10.198.3:41889)
0925 22:44:32.691728 (+ 899us) server_negotiation.cc:206] Beginning negotiation
0925 22:44:32.691731 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 22:44:32.691743 (+ 12us) server_negotiation.cc:407] Connection header received
0925 22:44:32.691813 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:44:32.691815 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:44:32.691912 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:44:32.691990 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:44:32.692442 (+ 452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:44:32.692943 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:44:32.694157 (+ 1214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:44:32.694747 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:44:32.694871 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:44:32.695209 (+ 338us) server_negotiation.cc:1035] Waiting for connection context
0925 22:44:32.695254 (+ 45us) server_negotiation.cc:299] Negotiation successful
0925 22:44:32.695315 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":817}
I20260925 22:44:32.696051 13060 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6cfaf161dbe7482baadac04f70a5453d" candidate_uuid: "3cad46077ef143feae0d7ee5ac6852f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a585751d44c49f98359ac6423e39c71" is_pre_election: true
I20260925 22:44:32.696142 13199 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6cfaf161dbe7482baadac04f70a5453d" candidate_uuid: "3cad46077ef143feae0d7ee5ac6852f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c1b2193f640945a895e6497c6422849f" is_pre_election: true
I20260925 22:44:32.696235 13060 raft_consensus.cc:2454] T 6cfaf161dbe7482baadac04f70a5453d P 2a585751d44c49f98359ac6423e39c71 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3cad46077ef143feae0d7ee5ac6852f4 in term 0.
I20260925 22:44:32.696311 13199 raft_consensus.cc:2454] T 6cfaf161dbe7482baadac04f70a5453d P c1b2193f640945a895e6497c6422849f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3cad46077ef143feae0d7ee5ac6852f4 in term 0.
I20260925 22:44:32.696693 12851 leader_election.cc:302] T 6cfaf161dbe7482baadac04f70a5453d P 3cad46077ef143feae0d7ee5ac6852f4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3cad46077ef143feae0d7ee5ac6852f4, c1b2193f640945a895e6497c6422849f; no voters:
I20260925 22:44:32.697026 13369 raft_consensus.cc:2781] T 6cfaf161dbe7482baadac04f70a5453d P 3cad46077ef143feae0d7ee5ac6852f4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 22:44:32.697124 13369 raft_consensus.cc:482] T 6cfaf161dbe7482baadac04f70a5453d P 3cad46077ef143feae0d7ee5ac6852f4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 22:44:32.697183 13369 raft_consensus.cc:3037] T 6cfaf161dbe7482baadac04f70a5453d P 3cad46077ef143feae0d7ee5ac6852f4 [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:44:32.698220 13369 raft_consensus.cc:504] T 6cfaf161dbe7482baadac04f70a5453d P 3cad46077ef143feae0d7ee5ac6852f4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cad46077ef143feae0d7ee5ac6852f4" member_type: VOTER last_known_addr { host: "127.10.198.1" port: 36901 } } peers { permanent_uuid: "2a585751d44c49f98359ac6423e39c71" member_type: VOTER last_known_addr { host: "127.10.198.2" port: 45425 } } peers { permanent_uuid: "c1b2193f640945a895e6497c6422849f" member_type: VOTER last_known_addr { host: "127.10.198.3" port: 41889 } }
I20260925 22:44:32.698705 13369 leader_election.cc:288] T 6cfaf161dbe7482baadac04f70a5453d P 3cad46077ef143feae0d7ee5ac6852f4 [CANDIDATE]: Term 1 election: Requested vote from peers 2a585751d44c49f98359ac6423e39c71 (127.10.198.2:45425), c1b2193f640945a895e6497c6422849f (127.10.198.3:41889)
I20260925 22:44:32.699108 13199 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6cfaf161dbe7482baadac04f70a5453d" candidate_uuid: "3cad46077ef143feae0d7ee5ac6852f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c1b2193f640945a895e6497c6422849f"
I20260925 22:44:32.699261 13199 raft_consensus.cc:3037] T 6cfaf161dbe7482baadac04f70a5453d P c1b2193f640945a895e6497c6422849f [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:44:32.700055 13060 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6cfaf161dbe7482baadac04f70a5453d" candidate_uuid: "3cad46077ef143feae0d7ee5ac6852f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a585751d44c49f98359ac6423e39c71"
I20260925 22:44:32.700217 13060 raft_consensus.cc:3037] T 6cfaf161dbe7482baadac04f70a5453d P 2a585751d44c49f98359ac6423e39c71 [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:44:32.700479 13199 raft_consensus.cc:2454] T 6cfaf161dbe7482baadac04f70a5453d P c1b2193f640945a895e6497c6422849f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3cad46077ef143feae0d7ee5ac6852f4 in term 1.
I20260925 22:44:32.700840 12851 leader_election.cc:302] T 6cfaf161dbe7482baadac04f70a5453d P 3cad46077ef143feae0d7ee5ac6852f4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3cad46077ef143feae0d7ee5ac6852f4, c1b2193f640945a895e6497c6422849f; no voters:
I20260925 22:44:32.701118 13369 raft_consensus.cc:2781] T 6cfaf161dbe7482baadac04f70a5453d P 3cad46077ef143feae0d7ee5ac6852f4 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 22:44:32.701380 13369 raft_consensus.cc:686] T 6cfaf161dbe7482baadac04f70a5453d P 3cad46077ef143feae0d7ee5ac6852f4 [term 1 LEADER]: Becoming Leader. State: Replica: 3cad46077ef143feae0d7ee5ac6852f4, State: Running, Role: LEADER
I20260925 22:44:32.701581 13060 raft_consensus.cc:2454] T 6cfaf161dbe7482baadac04f70a5453d P 2a585751d44c49f98359ac6423e39c71 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3cad46077ef143feae0d7ee5ac6852f4 in term 1.
I20260925 22:44:32.701944 13369 consensus_queue.cc:237] T 6cfaf161dbe7482baadac04f70a5453d P 3cad46077ef143feae0d7ee5ac6852f4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cad46077ef143feae0d7ee5ac6852f4" member_type: VOTER last_known_addr { host: "127.10.198.1" port: 36901 } } peers { permanent_uuid: "2a585751d44c49f98359ac6423e39c71" member_type: VOTER last_known_addr { host: "127.10.198.2" port: 45425 } } peers { permanent_uuid: "c1b2193f640945a895e6497c6422849f" member_type: VOTER last_known_addr { host: "127.10.198.3" port: 41889 } }
I20260925 22:44:32.704734 12761 catalog_manager.cc:5495] T 6cfaf161dbe7482baadac04f70a5453d P 3cad46077ef143feae0d7ee5ac6852f4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3cad46077ef143feae0d7ee5ac6852f4 (127.10.198.1). New cstate: current_term: 1 leader_uuid: "3cad46077ef143feae0d7ee5ac6852f4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cad46077ef143feae0d7ee5ac6852f4" member_type: VOTER last_known_addr { host: "127.10.198.1" port: 36901 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2a585751d44c49f98359ac6423e39c71" member_type: VOTER last_known_addr { host: "127.10.198.2" port: 45425 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c1b2193f640945a895e6497c6422849f" member_type: VOTER last_known_addr { host: "127.10.198.3" port: 41889 } health_report { overall_health: UNKNOWN } } }
I20260925 22:44:32.726073 13384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:44:32.721779 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45656 (local address 127.10.198.1:36901)
0925 22:44:32.722089 (+ 310us) server_negotiation.cc:206] Beginning negotiation
0925 22:44:32.722096 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 22:44:32.722109 (+ 13us) server_negotiation.cc:407] Connection header received
0925 22:44:32.722215 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:44:32.722218 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:44:32.722273 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:44:32.722424 (+ 151us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 22:44:32.722705 (+ 281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:44:32.723275 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:44:32.724225 (+ 950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:44:32.724487 (+ 262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:44:32.724607 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:44:32.724912 (+ 305us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 22:44:32.725408 (+ 496us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 22:44:32.725545 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0925 22:44:32.725636 (+ 91us) server_negotiation.cc:299] Negotiation successful
0925 22:44:32.725701 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":237,"thread_start_us":154,"threads_started":1}
I20260925 22:44:32.738103 13379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:44:32.734664 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.198.1:38925 (local address 127.10.198.2:45425)
0925 22:44:32.734792 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0925 22:44:32.734795 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 22:44:32.734877 (+ 82us) server_negotiation.cc:407] Connection header received
0925 22:44:32.735003 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:44:32.735005 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:44:32.735051 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:44:32.735129 (+ 78us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 22:44:32.735359 (+ 230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:44:32.735894 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:44:32.736959 (+ 1065us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:44:32.737209 (+ 250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:44:32.737356 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:44:32.737688 (+ 332us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 22:44:32.737770 (+ 82us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 22:44:32.737830 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0925 22:44:32.737928 (+ 98us) server_negotiation.cc:299] Negotiation successful
0925 22:44:32.737982 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260925 22:44:32.738131 13387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:44:32.734431 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.198.2:45425 (local address 127.10.198.1:38925)
0925 22:44:32.734795 (+ 364us) negotiation.cc:107] Waiting for socket to connect
0925 22:44:32.734810 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 22:44:32.734915 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:44:32.735156 (+ 241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:44:32.735158 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:44:32.735165 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 22:44:32.735271 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:44:32.735275 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:44:32.736068 (+ 793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:44:32.736074 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:44:32.736788 (+ 714us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:44:32.736801 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:44:32.737387 (+ 586us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:44:32.737389 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:44:32.737466 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:44:32.737551 (+ 85us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 22:44:32.737850 (+ 299us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 22:44:32.737855 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 22:44:32.737906 (+ 51us) client_negotiation.cc:240] Negotiation successful
0925 22:44:32.737985 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":251,"thread_start_us":131,"threads_started":1}
I20260925 22:44:32.744750 13199 raft_consensus.cc:1261] T 6cfaf161dbe7482baadac04f70a5453d P c1b2193f640945a895e6497c6422849f [term 1 FOLLOWER]: Refusing update from remote peer 3cad46077ef143feae0d7ee5ac6852f4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 22:44:32.744896 13060 raft_consensus.cc:1261] T 6cfaf161dbe7482baadac04f70a5453d P 2a585751d44c49f98359ac6423e39c71 [term 1 FOLLOWER]: Refusing update from remote peer 3cad46077ef143feae0d7ee5ac6852f4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 22:44:32.745322 13382 consensus_queue.cc:1035] T 6cfaf161dbe7482baadac04f70a5453d P 3cad46077ef143feae0d7ee5ac6852f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c1b2193f640945a895e6497c6422849f" member_type: VOTER last_known_addr { host: "127.10.198.3" port: 41889 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 22:44:32.745517 13369 consensus_queue.cc:1035] T 6cfaf161dbe7482baadac04f70a5453d P 3cad46077ef143feae0d7ee5ac6852f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a585751d44c49f98359ac6423e39c71" member_type: VOTER last_known_addr { host: "127.10.198.2" port: 45425 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 22:44:32.765244 13384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:44:32.761137 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.198.2:50193 (local address 127.10.198.1:36901)
0925 22:44:32.761399 (+ 262us) server_negotiation.cc:206] Beginning negotiation
0925 22:44:32.761504 (+ 105us) server_negotiation.cc:399] Waiting for connection header
0925 22:44:32.761523 (+ 19us) server_negotiation.cc:407] Connection header received
0925 22:44:32.761722 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:44:32.761728 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:44:32.761783 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:44:32.761857 (+ 74us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 22:44:32.762116 (+ 259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:44:32.762773 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:44:32.763961 (+ 1188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:44:32.764285 (+ 324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:44:32.764370 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:44:32.764714 (+ 344us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 22:44:32.764834 (+ 120us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 22:44:32.764917 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0925 22:44:32.765065 (+ 148us) server_negotiation.cc:299] Negotiation successful
0925 22:44:32.765126 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260925 22:44:32.765291 13392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:44:32.761069 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.198.1:36901 (local address 127.10.198.2:50193)
0925 22:44:32.761382 (+ 313us) negotiation.cc:107] Waiting for socket to connect
0925 22:44:32.761397 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 22:44:32.761576 (+ 179us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:44:32.761895 (+ 319us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:44:32.761900 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:44:32.761912 (+ 12us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 22:44:32.762035 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:44:32.762041 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:44:32.762885 (+ 844us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:44:32.762888 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:44:32.763773 (+ 885us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:44:32.763785 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:44:32.764401 (+ 616us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:44:32.764405 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:44:32.764515 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:44:32.764562 (+ 47us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 22:44:32.764945 (+ 383us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 22:44:32.764953 (+ 8us) client_negotiation.cc:769] Sending connection context
0925 22:44:32.765039 (+ 86us) client_negotiation.cc:240] Negotiation successful
0925 22:44:32.765135 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":210,"thread_start_us":129,"threads_started":1}
W20260925 22:44:32.766211 13332 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260925 22:44:34.331719 12761 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:36962:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 22:44:34.331921 12761 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:36962:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 22:44:34.334724 12761 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9f0e816c575a42caac240acda127e507: Sending DeleteTablet for 3 replicas of tablet 6cfaf161dbe7482baadac04f70a5453d
I20260925 22:44:34.335439 12899 tablet_service.cc:1514] Processing DeleteTablet for tablet 6cfaf161dbe7482baadac04f70a5453d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 22:44:34 UTC) from {username='kudu', principal='kudu/127.10.198.62@KRBTEST.COM'} at 127.0.0.1:38708
I20260925 22:44:34.335430 13180 tablet_service.cc:1514] Processing DeleteTablet for tablet 6cfaf161dbe7482baadac04f70a5453d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 22:44:34 UTC) from {username='kudu', principal='kudu/127.10.198.62@KRBTEST.COM'} at 127.0.0.1:45840
I20260925 22:44:34.335448 13040 tablet_service.cc:1514] Processing DeleteTablet for tablet 6cfaf161dbe7482baadac04f70a5453d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 22:44:34 UTC) from {username='kudu', principal='kudu/127.10.198.62@KRBTEST.COM'} at 127.0.0.1:33700
I20260925 22:44:34.335914 13419 tablet_replica.cc:330] T 6cfaf161dbe7482baadac04f70a5453d P 3cad46077ef143feae0d7ee5ac6852f4: stopping tablet replica
I20260925 22:44:34.335963 11032 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrWXm8X/build/debug/bin/kudu with pid 12835
I20260925 22:44:34.336903 13417 tablet_replica.cc:330] T 6cfaf161dbe7482baadac04f70a5453d P 2a585751d44c49f98359ac6423e39c71: stopping tablet replica
I20260925 22:44:34.337090 13417 raft_consensus.cc:2229] T 6cfaf161dbe7482baadac04f70a5453d P 2a585751d44c49f98359ac6423e39c71 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 22:44:34.337249 13417 raft_consensus.cc:2258] T 6cfaf161dbe7482baadac04f70a5453d P 2a585751d44c49f98359ac6423e39c71 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 22:44:34.337847 13418 tablet_replica.cc:330] T 6cfaf161dbe7482baadac04f70a5453d P c1b2193f640945a895e6497c6422849f: stopping tablet replica
I20260925 22:44:34.338009 13418 raft_consensus.cc:2229] T 6cfaf161dbe7482baadac04f70a5453d P c1b2193f640945a895e6497c6422849f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 22:44:34.338055 13417 ts_tablet_manager.cc:1905] T 6cfaf161dbe7482baadac04f70a5453d P 2a585751d44c49f98359ac6423e39c71: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 22:44:34.338162 13418 raft_consensus.cc:2258] T 6cfaf161dbe7482baadac04f70a5453d P c1b2193f640945a895e6497c6422849f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 22:44:34.340467 13418 ts_tablet_manager.cc:1905] T 6cfaf161dbe7482baadac04f70a5453d P c1b2193f640945a895e6497c6422849f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 22:44:34.341595 13417 ts_tablet_manager.cc:1918] T 6cfaf161dbe7482baadac04f70a5453d P 2a585751d44c49f98359ac6423e39c71: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260925 22:44:34.341696 13417 log.cc:1196] T 6cfaf161dbe7482baadac04f70a5453d P 2a585751d44c49f98359ac6423e39c71: Deleting WAL directory at /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790376259650642-11032-0/minicluster-data/ts-1/wal/wals/6cfaf161dbe7482baadac04f70a5453d
I20260925 22:44:34.342015 13417 ts_tablet_manager.cc:1939] T 6cfaf161dbe7482baadac04f70a5453d P 2a585751d44c49f98359ac6423e39c71: Deleting consensus metadata
I20260925 22:44:34.342891 13418 ts_tablet_manager.cc:1918] T 6cfaf161dbe7482baadac04f70a5453d P c1b2193f640945a895e6497c6422849f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260925 22:44:34.342980 13418 log.cc:1196] T 6cfaf161dbe7482baadac04f70a5453d P c1b2193f640945a895e6497c6422849f: Deleting WAL directory at /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790376259650642-11032-0/minicluster-data/ts-2/wal/wals/6cfaf161dbe7482baadac04f70a5453d
I20260925 22:44:34.342978 12748 catalog_manager.cc:4841] TS 2a585751d44c49f98359ac6423e39c71 (127.10.198.2:45425): tablet 6cfaf161dbe7482baadac04f70a5453d (table test-table [id=650c610e79a4447f80f75940f545196e]) successfully deleted
I20260925 22:44:34.343253 13418 ts_tablet_manager.cc:1939] T 6cfaf161dbe7482baadac04f70a5453d P c1b2193f640945a895e6497c6422849f: Deleting consensus metadata
I20260925 22:44:34.343992 12747 catalog_manager.cc:4841] TS c1b2193f640945a895e6497c6422849f (127.10.198.3:41889): tablet 6cfaf161dbe7482baadac04f70a5453d (table test-table [id=650c610e79a4447f80f75940f545196e]) successfully deleted
W20260925 22:44:34.347374 12749 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.10.198.1:36901 (error 108)
I20260925 22:44:34.347617 11032 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrWXm8X/build/debug/bin/kudu with pid 12975
I20260925 22:44:34.349234 13421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:44:34.348793 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.198.1:36901 (local address 127.0.0.1:45666)
0925 22:44:34.349071 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0925 22:44:34.349139 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.198.1:36901: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":202,"thread_start_us":102,"threads_started":1}
W20260925 22:44:34.349421 12749 catalog_manager.cc:4584] TS 3cad46077ef143feae0d7ee5ac6852f4 (127.10.198.1:36901): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 6cfaf161dbe7482baadac04f70a5453d: Network error: Client connection negotiation failed: client connection to 127.10.198.1:36901: connect: Connection refused (error 111)
I20260925 22:44:34.356910 11032 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrWXm8X/build/debug/bin/kudu with pid 13115
I20260925 22:44:34.364853 11032 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrWXm8X/build/debug/bin/kudu with pid 12731
2026-09-25T22:44:34Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (6466 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790376259650642-11032-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 22:44:34 dist-test-slave-5vdr krb5kdc[13424](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 22:44:34 dist-test-slave-5vdr krb5kdc[13424](info): set up 2 sockets
Sep 25 22:44:34 dist-test-slave-5vdr krb5kdc[13424](info): commencing operation
krb5kdc: starting...
W20260925 22:44:35.415879 11032 mini_kdc.cc:121] Time spent starting KDC: real 1.026s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790376259650642-11032-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790376259650642-11032-0/krb5kdc/test-user.keytab.
Sep 25 22:44:35 dist-test-slave-5vdr krb5kdc[13424](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376275, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T22:44:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T22:44:35Z Disabled control of system clock
WARNING: no policy specified for kudu/127.10.198.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.198.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.198.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790376259650642-11032-0/krb5kdc/kudu_127.10.198.62.keytab.
Entry for principal kudu/127.10.198.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790376259650642-11032-0/krb5kdc/kudu_127.10.198.62.keytab.
I20260925 22:44:35.551827 11032 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrWXm8X/build/debug/bin/kudu
/tmp/dist-test-taskrWXm8X/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790376259650642-11032-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790376259650642-11032-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790376259650642-11032-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790376259650642-11032-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.198.62:41385
--webserver_interface=127.10.198.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790376259650642-11032-0/krb5kdc/kudu_127.10.198.62.keytab
--principal=kudu/127.10.198.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.198.20:37433
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.198.62:41385
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790376259650642-11032-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790376259650642-11032-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790376259650642-11032-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:44:35.670760 13437 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:44:35.671142 13437 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:44:35.671237 13437 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:44:35.675595 13437 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 22:44:35.675732 13437 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:44:35.675769 13437 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:44:35.675794 13437 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 22:44:35.675822 13437 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 22:44:35.681772 13437 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.198.20:37433
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790376259650642-11032-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790376259650642-11032-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.198.62:41385
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790376259650642-11032-0/krb5kdc/kudu_127.10.198.62.keytab
--principal=kudu/127.10.198.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.198.62:41385
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790376259650642-11032-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.198.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13437
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790376259650642-11032-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:09:16 UTC on ccycloud.quasar-afmhyw.root.comops.site
build id 4673379
I20260925 22:44:35.683095 13437 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:44:35.684337 13437 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:44:35.694408 13443 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:44:35.694411 13445 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:44:35.694408 13442 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:44:35.694797 13437 server_base.cc:1034] running on GCE node
I20260925 22:44:35.695505 13437 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:44:35.696931 13437 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:44:35.698163 13437 hybrid_clock.cc:648] HybridClock initialized: now 1790376275698152 us; error 52 us; skew 500 ppm
Sep 25 22:44:35 dist-test-slave-5vdr krb5kdc[13424](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376275, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.198.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:44:35.701602 13437 init.cc:377] Logged in from keytab as kudu/127.10.198.62@KRBTEST.COM (short username kudu)
I20260925 22:44:35.702775 13437 webserver.cc:460] Webserver started at http://127.10.198.62:33727/ using document root <none> and password file <none>
I20260925 22:44:35.703485 13437 fs_manager.cc:359] Metadata directory not provided
I20260925 22:44:35.703620 13437 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:44:35.703962 13437 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:44:35.706301 13437 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790376259650642-11032-0/minicluster-data/master-0/data/instance:
uuid: "9a0344c1917d44e78ee5a3b52a235cc9"
format_stamp: "Formatted at 2026-09-25 22:44:35 on dist-test-slave-5vdr"
server_key: "7690346ec86c11230e149dd1d6eb5132"
server_key_iv: "ce1860c176296c48fb9e5cc467f502f4"
server_key_version: "encryptionkey@0"
I20260925 22:44:35.706979 13437 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790376259650642-11032-0/minicluster-data/master-0/wal/instance:
uuid: "9a0344c1917d44e78ee5a3b52a235cc9"
format_stamp: "Formatted at 2026-09-25 22:44:35 on dist-test-slave-5vdr"
server_key: "7690346ec86c11230e149dd1d6eb5132"
server_key_iv: "ce1860c176296c48fb9e5cc467f502f4"
server_key_version: "encryptionkey@0"
I20260925 22:44:35.711427 13437 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260925 22:44:35.714675 13452 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:44:35.716339 13437 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 22:44:35.716524 13437 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790376259650642-11032-0/minicluster-data/master-0/data,/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790376259650642-11032-0/minicluster-data/master-0/wal
uuid: "9a0344c1917d44e78ee5a3b52a235cc9"
format_stamp: "Formatted at 2026-09-25 22:44:35 on dist-test-slave-5vdr"
server_key: "7690346ec86c11230e149dd1d6eb5132"
server_key_iv: "ce1860c176296c48fb9e5cc467f502f4"
server_key_version: "encryptionkey@0"
I20260925 22:44:35.716674 13437 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790376259650642-11032-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790376259650642-11032-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790376259650642-11032-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:44:35.741856 13437 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:44:35.745486 13437 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:44:35.745735 13437 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:44:35.754213 13437 rpc_server.cc:307] RPC server started. Bound to: 127.10.198.62:41385
I20260925 22:44:35.754266 13504 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.198.62:41385 every 8 connection(s)
I20260925 22:44:35.755352 13437 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790376259650642-11032-0/minicluster-data/master-0/data/info.pb
I20260925 22:44:35.758736 13505 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:44:35.759707 11032 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrWXm8X/build/debug/bin/kudu as pid 13437
I20260925 22:44:35.759824 11032 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790376259650642-11032-0/minicluster-data/master-0/wal/instance
I20260925 22:44:35.760177 11032 external_mini_cluster.cc:1428] Setting key 5cba1e44e2463b09243eb7fbfcc17b18
I20260925 22:44:35.766846 13505 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9a0344c1917d44e78ee5a3b52a235cc9: Bootstrap starting.
I20260925 22:44:35.770005 13505 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9a0344c1917d44e78ee5a3b52a235cc9: Neither blocks nor log segments found. Creating new log.
I20260925 22:44:35.770936 13505 log.cc:824] T 00000000000000000000000000000000 P 9a0344c1917d44e78ee5a3b52a235cc9: Log is configured to *not* fsync() on all Append() calls
Sep 25 22:44:35 dist-test-slave-5vdr krb5kdc[13424](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376275, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.198.62@KRBTEST.COM
I20260925 22:44:35.773945 13505 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9a0344c1917d44e78ee5a3b52a235cc9: No bootstrap required, opened a new log
I20260925 22:44:35.778139 13508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:44:35.761865 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50940 (local address 127.10.198.62:41385)
0925 22:44:35.762392 (+ 527us) server_negotiation.cc:206] Beginning negotiation
0925 22:44:35.762416 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0925 22:44:35.762470 (+ 54us) server_negotiation.cc:407] Connection header received
0925 22:44:35.763405 (+ 935us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:44:35.763436 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:44:35.763976 (+ 540us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:44:35.764427 (+ 451us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:44:35.765226 (+ 799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:44:35.766780 (+ 1554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:44:35.767946 (+ 1166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:44:35.768745 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:44:35.768949 (+ 204us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:44:35.772044 (+ 3095us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:44:35.772053 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:44:35.772073 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:44:35.772104 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:44:35.774332 (+ 2228us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:44:35.774969 (+ 637us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:44:35.774975 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:44:35.774983 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:44:35.775143 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:44:35.775504 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:44:35.775509 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:44:35.775513 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:44:35.776152 (+ 639us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:44:35.776385 (+ 233us) server_negotiation.cc:1035] Waiting for connection context
0925 22:44:35.776677 (+ 292us) server_negotiation.cc:299] Negotiation successful
0925 22:44:35.777032 (+ 355us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":349,"thread_start_us":121,"threads_started":1}
I20260925 22:44:35.778878 13505 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9a0344c1917d44e78ee5a3b52a235cc9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a0344c1917d44e78ee5a3b52a235cc9" member_type: VOTER last_known_addr { host: "127.10.198.62" port: 41385 } }
I20260925 22:44:35.779182 13505 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9a0344c1917d44e78ee5a3b52a235cc9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:44:35.779259 13505 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9a0344c1917d44e78ee5a3b52a235cc9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a0344c1917d44e78ee5a3b52a235cc9, State: Initialized, Role: FOLLOWER
I20260925 22:44:35.779716 13505 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9a0344c1917d44e78ee5a3b52a235cc9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a0344c1917d44e78ee5a3b52a235cc9" member_type: VOTER last_known_addr { host: "127.10.198.62" port: 41385 } }
I20260925 22:44:35.779920 13505 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9a0344c1917d44e78ee5a3b52a235cc9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 22:44:35.779990 13505 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9a0344c1917d44e78ee5a3b52a235cc9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 22:44:35.780097 13505 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9a0344c1917d44e78ee5a3b52a235cc9 [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:44:35.781320 13505 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9a0344c1917d44e78ee5a3b52a235cc9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a0344c1917d44e78ee5a3b52a235cc9" member_type: VOTER last_known_addr { host: "127.10.198.62" port: 41385 } }
I20260925 22:44:35.781711 13505 leader_election.cc:302] T 00000000000000000000000000000000 P 9a0344c1917d44e78ee5a3b52a235cc9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9a0344c1917d44e78ee5a3b52a235cc9; no voters:
I20260925 22:44:35.782087 13505 leader_election.cc:288] T 00000000000000000000000000000000 P 9a0344c1917d44e78ee5a3b52a235cc9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 22:44:35.782245 13510 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9a0344c1917d44e78ee5a3b52a235cc9 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 22:44:35.782533 13510 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9a0344c1917d44e78ee5a3b52a235cc9 [term 1 LEADER]: Becoming Leader. State: Replica: 9a0344c1917d44e78ee5a3b52a235cc9, State: Running, Role: LEADER
I20260925 22:44:35.782873 13510 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9a0344c1917d44e78ee5a3b52a235cc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a0344c1917d44e78ee5a3b52a235cc9" member_type: VOTER last_known_addr { host: "127.10.198.62" port: 41385 } }
I20260925 22:44:35.783459 13505 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9a0344c1917d44e78ee5a3b52a235cc9 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 22:44:35.785003 13512 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9a0344c1917d44e78ee5a3b52a235cc9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9a0344c1917d44e78ee5a3b52a235cc9. Latest consensus state: current_term: 1 leader_uuid: "9a0344c1917d44e78ee5a3b52a235cc9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a0344c1917d44e78ee5a3b52a235cc9" member_type: VOTER last_known_addr { host: "127.10.198.62" port: 41385 } } }
I20260925 22:44:35.785526 13512 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9a0344c1917d44e78ee5a3b52a235cc9 [sys.catalog]: This master's current role is: LEADER
I20260925 22:44:35.785768 13511 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9a0344c1917d44e78ee5a3b52a235cc9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9a0344c1917d44e78ee5a3b52a235cc9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a0344c1917d44e78ee5a3b52a235cc9" member_type: VOTER last_known_addr { host: "127.10.198.62" port: 41385 } } }
I20260925 22:44:35.786110 13511 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9a0344c1917d44e78ee5a3b52a235cc9 [sys.catalog]: This master's current role is: LEADER
I20260925 22:44:35.786674 13525 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 22:44:35.789801 13525 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 22:44:35.796705 13525 catalog_manager.cc:1347] Generated new cluster ID: 0730467f4a564a1189215c0343acd047
I20260925 22:44:35.796800 13525 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 22:44:35.804755 13525 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 22:44:35.806174 13525 catalog_manager.cc:1504] Loading token signing keys...
I20260925 22:44:35.820308 13525 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9a0344c1917d44e78ee5a3b52a235cc9: Generated new TSK 0
I20260925 22:44:35.821292 13525 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.10.198.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.198.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.198.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790376259650642-11032-0/krb5kdc/kudu_127.10.198.1.keytab.
Entry for principal kudu/127.10.198.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790376259650642-11032-0/krb5kdc/kudu_127.10.198.1.keytab.
I20260925 22:44:35.851190 11032 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrWXm8X/build/debug/bin/kudu
/tmp/dist-test-taskrWXm8X/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790376259650642-11032-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790376259650642-11032-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790376259650642-11032-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790376259650642-11032-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.198.1:0
--local_ip_for_outbound_sockets=127.10.198.1
--webserver_interface=127.10.198.1
--webserver_port=0
--tserver_master_addrs=127.10.198.62:41385
--keytab_file=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790376259650642-11032-0/krb5kdc/kudu_127.10.198.1.keytab
--principal=kudu/127.10.198.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.198.20:37433
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790376259650642-11032-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790376259650642-11032-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790376259650642-11032-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:44:35.975257 13531 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:44:35.975586 13531 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:44:35.975633 13531 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:44:35.980202 13531 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:44:35.980293 13531 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:44:35.980360 13531 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.198.1
I20260925 22:44:35.986526 13531 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.198.20:37433
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790376259650642-11032-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790376259650642-11032-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790376259650642-11032-0/krb5kdc/kudu_127.10.198.1.keytab
--principal=kudu/127.10.198.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.198.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790376259650642-11032-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.198.1
--webserver_port=0
--tserver_master_addrs=127.10.198.62:41385
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13531
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.198.1
--log_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790376259650642-11032-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:09:16 UTC on ccycloud.quasar-afmhyw.root.comops.site
build id 4673379
I20260925 22:44:35.987869 13531 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:44:35.989084 13531 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:44:36.000339 13539 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:44:36.000347 13537 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:44:36.000397 13531 server_base.cc:1034] running on GCE node
W20260925 22:44:36.000335 13536 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:44:36.001087 13531 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:44:36.001860 13531 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:44:36.003086 13531 hybrid_clock.cc:648] HybridClock initialized: now 1790376276003058 us; error 45 us; skew 500 ppm
Sep 25 22:44:36 dist-test-slave-5vdr krb5kdc[13424](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376276, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.198.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:44:36.006354 13531 init.cc:377] Logged in from keytab as kudu/127.10.198.1@KRBTEST.COM (short username kudu)
I20260925 22:44:36.007426 13531 webserver.cc:460] Webserver started at http://127.10.198.1:35969/ using document root <none> and password file <none>
I20260925 22:44:36.008132 13531 fs_manager.cc:359] Metadata directory not provided
I20260925 22:44:36.008210 13531 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:44:36.008409 13531 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:44:36.010684 13531 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790376259650642-11032-0/minicluster-data/ts-0/data/instance:
uuid: "eb4ad0809bb94847a75a7c72f2fd6317"
format_stamp: "Formatted at 2026-09-25 22:44:36 on dist-test-slave-5vdr"
server_key: "8500067a7eb73931b02268b4416054fa"
server_key_iv: "78886c70c728d6e8d6e22213ceff226b"
server_key_version: "encryptionkey@0"
I20260925 22:44:36.011272 13531 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790376259650642-11032-0/minicluster-data/ts-0/wal/instance:
uuid: "eb4ad0809bb94847a75a7c72f2fd6317"
format_stamp: "Formatted at 2026-09-25 22:44:36 on dist-test-slave-5vdr"
server_key: "8500067a7eb73931b02268b4416054fa"
server_key_iv: "78886c70c728d6e8d6e22213ceff226b"
server_key_version: "encryptionkey@0"
I20260925 22:44:36.015690 13531 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260925 22:44:36.018679 13546 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:44:36.020308 13531 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 22:44:36.020447 13531 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790376259650642-11032-0/minicluster-data/ts-0/data,/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790376259650642-11032-0/minicluster-data/ts-0/wal
uuid: "eb4ad0809bb94847a75a7c72f2fd6317"
format_stamp: "Formatted at 2026-09-25 22:44:36 on dist-test-slave-5vdr"
server_key: "8500067a7eb73931b02268b4416054fa"
server_key_iv: "78886c70c728d6e8d6e22213ceff226b"
server_key_version: "encryptionkey@0"
I20260925 22:44:36.020619 13531 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790376259650642-11032-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790376259650642-11032-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790376259650642-11032-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:44:36.032545 13531 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:44:36.036413 13531 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:44:36.036599 13531 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:44:36.037194 13531 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 22:44:36.038151 13531 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:44:36.038209 13531 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:44:36.038255 13531 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:44:36.038271 13531 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 22:44:36.048084 13531 rpc_server.cc:307] RPC server started. Bound to: 127.10.198.1:37073
I20260925 22:44:36.048100 13659 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.198.1:37073 every 8 connection(s)
I20260925 22:44:36.049203 13531 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790376259650642-11032-0/minicluster-data/ts-0/data/info.pb
I20260925 22:44:36.058921 11032 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrWXm8X/build/debug/bin/kudu as pid 13531
I20260925 22:44:36.059067 11032 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790376259650642-11032-0/minicluster-data/ts-0/wal/instance
I20260925 22:44:36.059393 11032 external_mini_cluster.cc:1428] Setting key af2a2c50549d131b9a08429e6b4a7ed0
Sep 25 22:44:36 dist-test-slave-5vdr krb5kdc[13424](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376276, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.198.1@KRBTEST.COM for kudu/127.10.198.62@KRBTEST.COM
I20260925 22:44:36.065672 13508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:44:36.051392 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.198.1:49203 (local address 127.10.198.62:41385)
0925 22:44:36.051543 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0925 22:44:36.051546 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 22:44:36.052752 (+ 1206us) server_negotiation.cc:407] Connection header received
0925 22:44:36.053639 (+ 887us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:44:36.053642 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:44:36.053693 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:44:36.053780 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:44:36.055064 (+ 1284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:44:36.055609 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:44:36.057021 (+ 1412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:44:36.057263 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:44:36.057369 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:44:36.060547 (+ 3178us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:44:36.060551 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:44:36.060553 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:44:36.060580 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:44:36.062563 (+ 1983us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:44:36.063361 (+ 798us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:44:36.063376 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:44:36.063378 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:44:36.063506 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:44:36.064003 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:44:36.064009 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:44:36.064012 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:44:36.064299 (+ 287us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:44:36.064464 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0925 22:44:36.065086 (+ 622us) server_negotiation.cc:299] Negotiation successful
0925 22:44:36.065386 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260925 22:44:36.066691 13662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:44:36.051731 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.198.62:41385 (local address 127.10.198.1:49203)
0925 22:44:36.052545 (+ 814us) negotiation.cc:107] Waiting for socket to connect
0925 22:44:36.052592 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0925 22:44:36.053427 (+ 835us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:44:36.053931 (+ 504us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:44:36.053939 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:44:36.054341 (+ 402us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:44:36.054886 (+ 545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:44:36.054905 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:44:36.055770 (+ 865us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:44:36.055773 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:44:36.056867 (+ 1094us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:44:36.056881 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:44:36.057439 (+ 558us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:44:36.057443 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:44:36.057605 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:44:36.058384 (+ 779us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:44:36.058403 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:44:36.060326 (+ 1923us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:44:36.062763 (+ 2437us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:44:36.062770 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:44:36.062787 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:44:36.063201 (+ 414us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:44:36.063623 (+ 422us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:44:36.063625 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:44:36.063627 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:44:36.063813 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:44:36.064455 (+ 642us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:44:36.064481 (+ 26us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:44:36.064778 (+ 297us) client_negotiation.cc:769] Sending connection context
0925 22:44:36.064967 (+ 189us) client_negotiation.cc:240] Negotiation successful
0925 22:44:36.065513 (+ 546us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":606,"thread_start_us":138,"threads_started":1}
I20260925 22:44:36.068423 13660 heartbeater.cc:347] Connected to a master server at 127.10.198.62:41385
I20260925 22:44:36.068784 13660 heartbeater.cc:464] Registering TS with master...
I20260925 22:44:36.069581 13660 heartbeater.cc:511] Master 127.10.198.62:41385 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.10.198.2@KRBTEST.COM; defaulting to no policy
I20260925 22:44:36.071741 13469 ts_manager.cc:194] Registered new tserver with Master: eb4ad0809bb94847a75a7c72f2fd6317 (127.10.198.1:37073)
I20260925 22:44:36.073082 13469 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.198.1@KRBTEST.COM'} at 127.10.198.1:49203
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.198.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.198.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790376259650642-11032-0/krb5kdc/kudu_127.10.198.2.keytab.
Entry for principal kudu/127.10.198.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790376259650642-11032-0/krb5kdc/kudu_127.10.198.2.keytab.
I20260925 22:44:36.092283 11032 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrWXm8X/build/debug/bin/kudu
/tmp/dist-test-taskrWXm8X/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790376259650642-11032-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790376259650642-11032-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790376259650642-11032-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790376259650642-11032-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.198.2:0
--local_ip_for_outbound_sockets=127.10.198.2
--webserver_interface=127.10.198.2
--webserver_port=0
--tserver_master_addrs=127.10.198.62:41385
--keytab_file=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790376259650642-11032-0/krb5kdc/kudu_127.10.198.2.keytab
--principal=kudu/127.10.198.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.198.20:37433
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790376259650642-11032-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790376259650642-11032-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790376259650642-11032-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:44:36.216663 13665 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:44:36.217053 13665 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:44:36.217109 13665 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:44:36.222736 13665 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:44:36.222872 13665 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:44:36.222949 13665 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.198.2
I20260925 22:44:36.230217 13665 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.198.20:37433
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790376259650642-11032-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790376259650642-11032-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790376259650642-11032-0/krb5kdc/kudu_127.10.198.2.keytab
--principal=kudu/127.10.198.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.198.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790376259650642-11032-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.198.2
--webserver_port=0
--tserver_master_addrs=127.10.198.62:41385
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13665
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.198.2
--log_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790376259650642-11032-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:09:16 UTC on ccycloud.quasar-afmhyw.root.comops.site
build id 4673379
I20260925 22:44:36.231465 13665 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:44:36.232688 13665 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:44:36.243609 13670 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:44:36.243609 13673 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:44:36.243630 13671 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:44:36.244009 13665 server_base.cc:1034] running on GCE node
I20260925 22:44:36.244628 13665 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:44:36.245440 13665 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:44:36.246696 13665 hybrid_clock.cc:648] HybridClock initialized: now 1790376276246675 us; error 42 us; skew 500 ppm
Sep 25 22:44:36 dist-test-slave-5vdr krb5kdc[13424](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376276, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.198.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:44:36.250017 13665 init.cc:377] Logged in from keytab as kudu/127.10.198.2@KRBTEST.COM (short username kudu)
I20260925 22:44:36.251147 13665 webserver.cc:460] Webserver started at http://127.10.198.2:38877/ using document root <none> and password file <none>
I20260925 22:44:36.251747 13665 fs_manager.cc:359] Metadata directory not provided
I20260925 22:44:36.251816 13665 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:44:36.252072 13665 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:44:36.254261 13665 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790376259650642-11032-0/minicluster-data/ts-1/data/instance:
uuid: "30732f36a887455293447028d696d3db"
format_stamp: "Formatted at 2026-09-25 22:44:36 on dist-test-slave-5vdr"
server_key: "d269b2b6783903076b1965f3b6b94da0"
server_key_iv: "7464301f861495331c9b95b145af5b56"
server_key_version: "encryptionkey@0"
I20260925 22:44:36.254840 13665 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790376259650642-11032-0/minicluster-data/ts-1/wal/instance:
uuid: "30732f36a887455293447028d696d3db"
format_stamp: "Formatted at 2026-09-25 22:44:36 on dist-test-slave-5vdr"
server_key: "d269b2b6783903076b1965f3b6b94da0"
server_key_iv: "7464301f861495331c9b95b145af5b56"
server_key_version: "encryptionkey@0"
I20260925 22:44:36.259087 13665 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260925 22:44:36.262097 13680 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:44:36.263682 13665 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260925 22:44:36.263813 13665 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790376259650642-11032-0/minicluster-data/ts-1/data,/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790376259650642-11032-0/minicluster-data/ts-1/wal
uuid: "30732f36a887455293447028d696d3db"
format_stamp: "Formatted at 2026-09-25 22:44:36 on dist-test-slave-5vdr"
server_key: "d269b2b6783903076b1965f3b6b94da0"
server_key_iv: "7464301f861495331c9b95b145af5b56"
server_key_version: "encryptionkey@0"
I20260925 22:44:36.264032 13665 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790376259650642-11032-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790376259650642-11032-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790376259650642-11032-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:44:36.285177 13665 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:44:36.289273 13665 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:44:36.289518 13665 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:44:36.290220 13665 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 22:44:36.291211 13665 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:44:36.291271 13665 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:44:36.291320 13665 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:44:36.291339 13665 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 22:44:36.302223 13665 rpc_server.cc:307] RPC server started. Bound to: 127.10.198.2:42885
I20260925 22:44:36.302300 13793 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.198.2:42885 every 8 connection(s)
I20260925 22:44:36.303620 13665 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790376259650642-11032-0/minicluster-data/ts-1/data/info.pb
I20260925 22:44:36.310176 11032 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrWXm8X/build/debug/bin/kudu as pid 13665
I20260925 22:44:36.310309 11032 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790376259650642-11032-0/minicluster-data/ts-1/wal/instance
I20260925 22:44:36.310689 11032 external_mini_cluster.cc:1428] Setting key f843989c5213292d41334fd99c93678a
Sep 25 22:44:36 dist-test-slave-5vdr krb5kdc[13424](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376276, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.198.2@KRBTEST.COM for kudu/127.10.198.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.10.198.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.198.3@KRBTEST.COM" created.
I20260925 22:44:36.329078 13508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:44:36.311185 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.198.2:47261 (local address 127.10.198.62:41385)
0925 22:44:36.311360 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0925 22:44:36.311366 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 22:44:36.312909 (+ 1543us) server_negotiation.cc:407] Connection header received
0925 22:44:36.314444 (+ 1535us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:44:36.314450 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:44:36.314526 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:44:36.314633 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:44:36.316083 (+ 1450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:44:36.316653 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:44:36.319001 (+ 2348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:44:36.319588 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:44:36.319749 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:44:36.323259 (+ 3510us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:44:36.323263 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:44:36.323265 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:44:36.323292 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:44:36.325349 (+ 2057us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:44:36.326205 (+ 856us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:44:36.326209 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:44:36.326211 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:44:36.326323 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:44:36.327187 (+ 864us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:44:36.327192 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:44:36.327194 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:44:36.327504 (+ 310us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:44:36.327660 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0925 22:44:36.328700 (+ 1040us) server_negotiation.cc:299] Negotiation successful
0925 22:44:36.328884 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260925 22:44:36.330190 13796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:44:36.311741 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.198.62:41385 (local address 127.10.198.2:47261)
0925 22:44:36.312722 (+ 981us) negotiation.cc:107] Waiting for socket to connect
0925 22:44:36.312792 (+ 70us) client_negotiation.cc:174] Beginning negotiation
0925 22:44:36.314124 (+ 1332us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:44:36.314823 (+ 699us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:44:36.314831 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:44:36.315265 (+ 434us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:44:36.315838 (+ 573us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:44:36.315926 (+ 88us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:44:36.316811 (+ 885us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:44:36.316818 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:44:36.318773 (+ 1955us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:44:36.318796 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:44:36.319826 (+ 1030us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:44:36.319833 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:44:36.320155 (+ 322us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:44:36.320965 (+ 810us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:44:36.320998 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:44:36.323024 (+ 2026us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:44:36.325522 (+ 2498us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:44:36.325531 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:44:36.325560 (+ 29us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:44:36.326046 (+ 486us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:44:36.326448 (+ 402us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:44:36.326451 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:44:36.326453 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:44:36.326934 (+ 481us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:44:36.327657 (+ 723us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:44:36.327670 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:44:36.328302 (+ 632us) client_negotiation.cc:769] Sending connection context
0925 22:44:36.328592 (+ 290us) client_negotiation.cc:240] Negotiation successful
0925 22:44:36.329013 (+ 421us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":709,"thread_start_us":150,"threads_started":1}
I20260925 22:44:36.331892 13794 heartbeater.cc:347] Connected to a master server at 127.10.198.62:41385
I20260925 22:44:36.332224 13794 heartbeater.cc:464] Registering TS with master...
I20260925 22:44:36.333033 13794 heartbeater.cc:511] Master 127.10.198.62:41385 requested a full tablet report, sending...
I20260925 22:44:36.334501 13469 ts_manager.cc:194] Registered new tserver with Master: 30732f36a887455293447028d696d3db (127.10.198.2:42885)
I20260925 22:44:36.335436 13469 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.198.2@KRBTEST.COM'} at 127.10.198.2:47261
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.198.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790376259650642-11032-0/krb5kdc/kudu_127.10.198.3.keytab.
Entry for principal kudu/127.10.198.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790376259650642-11032-0/krb5kdc/kudu_127.10.198.3.keytab.
I20260925 22:44:36.344107 11032 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrWXm8X/build/debug/bin/kudu
/tmp/dist-test-taskrWXm8X/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790376259650642-11032-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790376259650642-11032-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790376259650642-11032-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790376259650642-11032-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.198.3:0
--local_ip_for_outbound_sockets=127.10.198.3
--webserver_interface=127.10.198.3
--webserver_port=0
--tserver_master_addrs=127.10.198.62:41385
--keytab_file=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790376259650642-11032-0/krb5kdc/kudu_127.10.198.3.keytab
--principal=kudu/127.10.198.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.198.20:37433
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790376259650642-11032-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790376259650642-11032-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790376259650642-11032-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:44:36.465703 13799 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:44:36.466043 13799 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:44:36.466102 13799 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:44:36.470710 13799 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:44:36.470805 13799 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:44:36.470932 13799 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.198.3
I20260925 22:44:36.476816 13799 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.198.20:37433
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790376259650642-11032-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790376259650642-11032-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790376259650642-11032-0/krb5kdc/kudu_127.10.198.3.keytab
--principal=kudu/127.10.198.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.198.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790376259650642-11032-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.198.3
--webserver_port=0
--tserver_master_addrs=127.10.198.62:41385
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13799
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.198.3
--log_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790376259650642-11032-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:09:16 UTC on ccycloud.quasar-afmhyw.root.comops.site
build id 4673379
I20260925 22:44:36.478072 13799 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:44:36.479259 13799 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:44:36.490424 13805 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:44:36.490409 13807 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:44:36.490433 13804 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:44:36.490944 13799 server_base.cc:1034] running on GCE node
I20260925 22:44:36.491441 13799 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:44:36.492206 13799 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:44:36.493376 13799 hybrid_clock.cc:648] HybridClock initialized: now 1790376276493337 us; error 58 us; skew 500 ppm
Sep 25 22:44:36 dist-test-slave-5vdr krb5kdc[13424](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376276, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.198.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:44:36.496605 13799 init.cc:377] Logged in from keytab as kudu/127.10.198.3@KRBTEST.COM (short username kudu)
I20260925 22:44:36.497756 13799 webserver.cc:460] Webserver started at http://127.10.198.3:43713/ using document root <none> and password file <none>
I20260925 22:44:36.498430 13799 fs_manager.cc:359] Metadata directory not provided
I20260925 22:44:36.498528 13799 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:44:36.498775 13799 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:44:36.501101 13799 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790376259650642-11032-0/minicluster-data/ts-2/data/instance:
uuid: "c4428f4aaf294046835acd416efc08e2"
format_stamp: "Formatted at 2026-09-25 22:44:36 on dist-test-slave-5vdr"
server_key: "79b1abe53ea49d03575d6ee8c00474ae"
server_key_iv: "a5b2c80734ad0564639ee1b30a68ec0d"
server_key_version: "encryptionkey@0"
I20260925 22:44:36.501713 13799 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790376259650642-11032-0/minicluster-data/ts-2/wal/instance:
uuid: "c4428f4aaf294046835acd416efc08e2"
format_stamp: "Formatted at 2026-09-25 22:44:36 on dist-test-slave-5vdr"
server_key: "79b1abe53ea49d03575d6ee8c00474ae"
server_key_iv: "a5b2c80734ad0564639ee1b30a68ec0d"
server_key_version: "encryptionkey@0"
I20260925 22:44:36.506078 13799 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260925 22:44:36.509006 13814 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:44:36.510506 13799 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260925 22:44:36.510699 13799 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790376259650642-11032-0/minicluster-data/ts-2/data,/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790376259650642-11032-0/minicluster-data/ts-2/wal
uuid: "c4428f4aaf294046835acd416efc08e2"
format_stamp: "Formatted at 2026-09-25 22:44:36 on dist-test-slave-5vdr"
server_key: "79b1abe53ea49d03575d6ee8c00474ae"
server_key_iv: "a5b2c80734ad0564639ee1b30a68ec0d"
server_key_version: "encryptionkey@0"
I20260925 22:44:36.510843 13799 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790376259650642-11032-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790376259650642-11032-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790376259650642-11032-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:44:36.538221 13799 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:44:36.541926 13799 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:44:36.542176 13799 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:44:36.542850 13799 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 22:44:36.543787 13799 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:44:36.543890 13799 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:44:36.543985 13799 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:44:36.544035 13799 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 22:44:36.553789 13799 rpc_server.cc:307] RPC server started. Bound to: 127.10.198.3:38949
I20260925 22:44:36.553817 13927 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.198.3:38949 every 8 connection(s)
I20260925 22:44:36.554782 13799 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790376259650642-11032-0/minicluster-data/ts-2/data/info.pb
I20260925 22:44:36.562495 11032 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrWXm8X/build/debug/bin/kudu as pid 13799
I20260925 22:44:36.562629 11032 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790376259650642-11032-0/minicluster-data/ts-2/wal/instance
I20260925 22:44:36.562973 11032 external_mini_cluster.cc:1428] Setting key 539b81cf148eb7297d7744c2ea2e5e84
Sep 25 22:44:36 dist-test-slave-5vdr krb5kdc[13424](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376276, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.198.3@KRBTEST.COM for kudu/127.10.198.62@KRBTEST.COM
I20260925 22:44:36.571044 13508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:44:36.556935 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.198.3:37589 (local address 127.10.198.62:41385)
0925 22:44:36.557087 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0925 22:44:36.557090 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 22:44:36.558264 (+ 1174us) server_negotiation.cc:407] Connection header received
0925 22:44:36.559255 (+ 991us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:44:36.559259 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:44:36.559323 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:44:36.559431 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:44:36.560789 (+ 1358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:44:36.561333 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:44:36.562792 (+ 1459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:44:36.563041 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:44:36.563178 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:44:36.566301 (+ 3123us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:44:36.566305 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:44:36.566307 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:44:36.566334 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:44:36.568251 (+ 1917us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:44:36.568982 (+ 731us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:44:36.568986 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:44:36.568987 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:44:36.569147 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:44:36.569664 (+ 517us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:44:36.569667 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:44:36.569669 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:44:36.569902 (+ 233us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:44:36.569997 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0925 22:44:36.570759 (+ 762us) server_negotiation.cc:299] Negotiation successful
0925 22:44:36.570912 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260925 22:44:36.572223 13930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:44:36.557388 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.198.62:41385 (local address 127.10.198.3:37589)
0925 22:44:36.558087 (+ 699us) negotiation.cc:107] Waiting for socket to connect
0925 22:44:36.558143 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0925 22:44:36.559012 (+ 869us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:44:36.559622 (+ 610us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:44:36.559631 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:44:36.560097 (+ 466us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:44:36.560625 (+ 528us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:44:36.560641 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:44:36.561480 (+ 839us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:44:36.561484 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:44:36.562638 (+ 1154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:44:36.562654 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:44:36.563267 (+ 613us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:44:36.563272 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:44:36.563498 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:44:36.564285 (+ 787us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:44:36.564305 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:44:36.566055 (+ 1750us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:44:36.568478 (+ 2423us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:44:36.568485 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:44:36.568503 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:44:36.568827 (+ 324us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:44:36.569348 (+ 521us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:44:36.569351 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:44:36.569352 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:44:36.569539 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:44:36.570036 (+ 497us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:44:36.570043 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:44:36.570400 (+ 357us) client_negotiation.cc:769] Sending connection context
0925 22:44:36.570592 (+ 192us) client_negotiation.cc:240] Negotiation successful
0925 22:44:36.570932 (+ 340us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":497,"thread_start_us":125,"threads_started":1}
I20260925 22:44:36.573673 13928 heartbeater.cc:347] Connected to a master server at 127.10.198.62:41385
I20260925 22:44:36.573926 13928 heartbeater.cc:464] Registering TS with master...
I20260925 22:44:36.574573 13928 heartbeater.cc:511] Master 127.10.198.62:41385 requested a full tablet report, sending...
I20260925 22:44:36.575963 13469 ts_manager.cc:194] Registered new tserver with Master: c4428f4aaf294046835acd416efc08e2 (127.10.198.3:38949)
I20260925 22:44:36.576803 13469 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.198.3@KRBTEST.COM'} at 127.10.198.3:37589
I20260925 22:44:36.577922 11032 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260925 22:44:36.594991 13508 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:44:36.593166 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50948 (local address 127.10.198.62:41385)
0925 22:44:36.593384 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0925 22:44:36.593389 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:44:36.593597 (+ 208us) server_negotiation.cc:407] Connection header received
0925 22:44:36.593800 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:44:36.593804 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:44:36.593849 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:44:36.593944 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:44:36.594829 (+ 885us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50948: BlockingRecv error: recv got EOF from 127.0.0.1:50948 (error 108)
Metrics: {"server-negotiator.queue_time_us":84}
I20260925 22:44:36.596455 11032 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrWXm8X/build/debug/bin/kudu with pid 13531
I20260925 22:44:36.602167 11032 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrWXm8X/build/debug/bin/kudu with pid 13665
I20260925 22:44:36.607658 11032 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrWXm8X/build/debug/bin/kudu with pid 13799
I20260925 22:44:36.613430 11032 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrWXm8X/build/debug/bin/kudu with pid 13437
2026-09-25T22:44:36Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (2246 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790376259650642-11032-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 22:44:36 dist-test-slave-5vdr krb5kdc[13940](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 22:44:36 dist-test-slave-5vdr krb5kdc[13940](info): set up 2 sockets
Sep 25 22:44:36 dist-test-slave-5vdr krb5kdc[13940](info): commencing operation
krb5kdc: starting...
W20260925 22:44:37.672233 11032 mini_kdc.cc:121] Time spent starting KDC: real 1.036s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790376259650642-11032-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790376259650642-11032-0/krb5kdc/test-user.keytab.
Sep 25 22:44:37 dist-test-slave-5vdr krb5kdc[13940](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376277, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T22:44:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T22:44:37Z Disabled control of system clock
WARNING: no policy specified for kudu/127.10.198.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.198.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.198.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790376259650642-11032-0/krb5kdc/kudu_127.10.198.62.keytab.
Entry for principal kudu/127.10.198.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790376259650642-11032-0/krb5kdc/kudu_127.10.198.62.keytab.
I20260925 22:44:37.803341 11032 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrWXm8X/build/debug/bin/kudu
/tmp/dist-test-taskrWXm8X/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790376259650642-11032-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790376259650642-11032-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790376259650642-11032-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790376259650642-11032-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.198.62:36987
--webserver_interface=127.10.198.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790376259650642-11032-0/krb5kdc/kudu_127.10.198.62.keytab
--principal=kudu/127.10.198.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.198.20:38479
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.198.62:36987
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790376259650642-11032-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790376259650642-11032-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790376259650642-11032-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:44:37.920931 13953 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:44:37.921250 13953 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:44:37.921293 13953 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:44:37.925698 13953 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 22:44:37.925783 13953 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:44:37.925804 13953 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:44:37.925822 13953 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 22:44:37.925837 13953 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 22:44:37.931689 13953 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.198.20:38479
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790376259650642-11032-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790376259650642-11032-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.198.62:36987
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790376259650642-11032-0/krb5kdc/kudu_127.10.198.62.keytab
--principal=kudu/127.10.198.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.198.62:36987
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790376259650642-11032-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.198.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13953
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790376259650642-11032-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:09:16 UTC on ccycloud.quasar-afmhyw.root.comops.site
build id 4673379
I20260925 22:44:37.933007 13953 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:44:37.934158 13953 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:44:37.943482 13961 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:44:37.943490 13958 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:44:37.943491 13959 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:44:37.943958 13953 server_base.cc:1034] running on GCE node
I20260925 22:44:37.944666 13953 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:44:37.946074 13953 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:44:37.947320 13953 hybrid_clock.cc:648] HybridClock initialized: now 1790376277947307 us; error 35 us; skew 500 ppm
Sep 25 22:44:37 dist-test-slave-5vdr krb5kdc[13940](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376277, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.198.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:44:37.958841 13953 init.cc:377] Logged in from keytab as kudu/127.10.198.62@KRBTEST.COM (short username kudu)
I20260925 22:44:37.960059 13953 webserver.cc:460] Webserver started at http://127.10.198.62:37781/ using document root <none> and password file <none>
I20260925 22:44:37.960672 13953 fs_manager.cc:359] Metadata directory not provided
I20260925 22:44:37.960745 13953 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:44:37.960927 13953 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:44:37.963156 13953 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790376259650642-11032-0/minicluster-data/master-0/data/instance:
uuid: "a0ea264bfe704cf1b386e5fefb6805d8"
format_stamp: "Formatted at 2026-09-25 22:44:37 on dist-test-slave-5vdr"
server_key: "48246430112ba593d11e4c1bfb09daff"
server_key_iv: "23bcb77d97a08048e8927eb1f9367814"
server_key_version: "encryptionkey@0"
I20260925 22:44:37.963750 13953 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790376259650642-11032-0/minicluster-data/master-0/wal/instance:
uuid: "a0ea264bfe704cf1b386e5fefb6805d8"
format_stamp: "Formatted at 2026-09-25 22:44:37 on dist-test-slave-5vdr"
server_key: "48246430112ba593d11e4c1bfb09daff"
server_key_iv: "23bcb77d97a08048e8927eb1f9367814"
server_key_version: "encryptionkey@0"
I20260925 22:44:37.968124 13953 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260925 22:44:37.971065 13968 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:44:37.972752 13953 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 22:44:37.972898 13953 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790376259650642-11032-0/minicluster-data/master-0/data,/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790376259650642-11032-0/minicluster-data/master-0/wal
uuid: "a0ea264bfe704cf1b386e5fefb6805d8"
format_stamp: "Formatted at 2026-09-25 22:44:37 on dist-test-slave-5vdr"
server_key: "48246430112ba593d11e4c1bfb09daff"
server_key_iv: "23bcb77d97a08048e8927eb1f9367814"
server_key_version: "encryptionkey@0"
I20260925 22:44:37.973012 13953 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790376259650642-11032-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790376259650642-11032-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790376259650642-11032-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:44:37.986903 13953 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:44:37.990511 13953 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:44:37.990700 13953 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:44:37.998741 13953 rpc_server.cc:307] RPC server started. Bound to: 127.10.198.62:36987
I20260925 22:44:37.999065 14020 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.198.62:36987 every 8 connection(s)
I20260925 22:44:38.000046 13953 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790376259650642-11032-0/minicluster-data/master-0/data/info.pb
I20260925 22:44:38.000837 11032 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrWXm8X/build/debug/bin/kudu as pid 13953
I20260925 22:44:38.000978 11032 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790376259650642-11032-0/minicluster-data/master-0/wal/instance
I20260925 22:44:38.001289 11032 external_mini_cluster.cc:1428] Setting key 620e4e1a3b018fb9fb346631d123f0d5
I20260925 22:44:38.003911 14021 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:44:38.010392 14021 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a0ea264bfe704cf1b386e5fefb6805d8: Bootstrap starting.
Sep 25 22:44:38 dist-test-slave-5vdr krb5kdc[13940](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376277, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.198.62@KRBTEST.COM
I20260925 22:44:38.013294 14021 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a0ea264bfe704cf1b386e5fefb6805d8: Neither blocks nor log segments found. Creating new log.
I20260925 22:44:38.014182 14021 log.cc:824] T 00000000000000000000000000000000 P a0ea264bfe704cf1b386e5fefb6805d8: Log is configured to *not* fsync() on all Append() calls
I20260925 22:44:38.017019 14021 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a0ea264bfe704cf1b386e5fefb6805d8: No bootstrap required, opened a new log
I20260925 22:44:38.017596 14024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:44:38.002912 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60182 (local address 127.10.198.62:36987)
0925 22:44:38.003451 (+ 539us) server_negotiation.cc:206] Beginning negotiation
0925 22:44:38.003468 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0925 22:44:38.003509 (+ 41us) server_negotiation.cc:407] Connection header received
0925 22:44:38.004353 (+ 844us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:44:38.004376 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:44:38.004757 (+ 381us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:44:38.005151 (+ 394us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:44:38.006076 (+ 925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:44:38.007138 (+ 1062us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:44:38.008203 (+ 1065us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:44:38.008743 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:44:38.008952 (+ 209us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:44:38.011783 (+ 2831us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:44:38.011789 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:44:38.011805 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:44:38.011900 (+ 95us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:44:38.013890 (+ 1990us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:44:38.014454 (+ 564us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:44:38.014461 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:44:38.014468 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:44:38.014649 (+ 181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:44:38.015008 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:44:38.015011 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:44:38.015012 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:44:38.015579 (+ 567us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:44:38.015816 (+ 237us) server_negotiation.cc:1035] Waiting for connection context
0925 22:44:38.016176 (+ 360us) server_negotiation.cc:299] Negotiation successful
0925 22:44:38.016530 (+ 354us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":359,"thread_start_us":168,"threads_started":1}
I20260925 22:44:38.021003 11032 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 22:44:38.021759 14021 raft_consensus.cc:348] T 00000000000000000000000000000000 P a0ea264bfe704cf1b386e5fefb6805d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a0ea264bfe704cf1b386e5fefb6805d8" member_type: VOTER last_known_addr { host: "127.10.198.62" port: 36987 } }
I20260925 22:44:38.022033 14021 raft_consensus.cc:374] T 00000000000000000000000000000000 P a0ea264bfe704cf1b386e5fefb6805d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:44:38.022110 14021 raft_consensus.cc:729] T 00000000000000000000000000000000 P a0ea264bfe704cf1b386e5fefb6805d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a0ea264bfe704cf1b386e5fefb6805d8, State: Initialized, Role: FOLLOWER
I20260925 22:44:38.022612 14021 consensus_queue.cc:260] T 00000000000000000000000000000000 P a0ea264bfe704cf1b386e5fefb6805d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a0ea264bfe704cf1b386e5fefb6805d8" member_type: VOTER last_known_addr { host: "127.10.198.62" port: 36987 } }
I20260925 22:44:38.022789 14021 raft_consensus.cc:388] T 00000000000000000000000000000000 P a0ea264bfe704cf1b386e5fefb6805d8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 22:44:38.022852 14021 raft_consensus.cc:482] T 00000000000000000000000000000000 P a0ea264bfe704cf1b386e5fefb6805d8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 22:44:38.023002 14021 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a0ea264bfe704cf1b386e5fefb6805d8 [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:44:38.024139 14021 raft_consensus.cc:504] T 00000000000000000000000000000000 P a0ea264bfe704cf1b386e5fefb6805d8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a0ea264bfe704cf1b386e5fefb6805d8" member_type: VOTER last_known_addr { host: "127.10.198.62" port: 36987 } }
I20260925 22:44:38.024540 14021 leader_election.cc:302] T 00000000000000000000000000000000 P a0ea264bfe704cf1b386e5fefb6805d8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a0ea264bfe704cf1b386e5fefb6805d8; no voters:
I20260925 22:44:38.024950 14021 leader_election.cc:288] T 00000000000000000000000000000000 P a0ea264bfe704cf1b386e5fefb6805d8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 22:44:38.025020 14027 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a0ea264bfe704cf1b386e5fefb6805d8 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 22:44:38.025233 14027 raft_consensus.cc:686] T 00000000000000000000000000000000 P a0ea264bfe704cf1b386e5fefb6805d8 [term 1 LEADER]: Becoming Leader. State: Replica: a0ea264bfe704cf1b386e5fefb6805d8, State: Running, Role: LEADER
I20260925 22:44:38.025559 14027 consensus_queue.cc:237] T 00000000000000000000000000000000 P a0ea264bfe704cf1b386e5fefb6805d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a0ea264bfe704cf1b386e5fefb6805d8" member_type: VOTER last_known_addr { host: "127.10.198.62" port: 36987 } }
I20260925 22:44:38.026417 14021 sys_catalog.cc:564] T 00000000000000000000000000000000 P a0ea264bfe704cf1b386e5fefb6805d8 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 22:44:38.028656 14029 sys_catalog.cc:455] T 00000000000000000000000000000000 P a0ea264bfe704cf1b386e5fefb6805d8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a0ea264bfe704cf1b386e5fefb6805d8. Latest consensus state: current_term: 1 leader_uuid: "a0ea264bfe704cf1b386e5fefb6805d8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a0ea264bfe704cf1b386e5fefb6805d8" member_type: VOTER last_known_addr { host: "127.10.198.62" port: 36987 } } }
I20260925 22:44:38.028656 14028 sys_catalog.cc:455] T 00000000000000000000000000000000 P a0ea264bfe704cf1b386e5fefb6805d8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a0ea264bfe704cf1b386e5fefb6805d8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a0ea264bfe704cf1b386e5fefb6805d8" member_type: VOTER last_known_addr { host: "127.10.198.62" port: 36987 } } }
I20260925 22:44:38.029220 14028 sys_catalog.cc:458] T 00000000000000000000000000000000 P a0ea264bfe704cf1b386e5fefb6805d8 [sys.catalog]: This master's current role is: LEADER
I20260925 22:44:38.029435 14029 sys_catalog.cc:458] T 00000000000000000000000000000000 P a0ea264bfe704cf1b386e5fefb6805d8 [sys.catalog]: This master's current role is: LEADER
W20260925 22:44:38.032182 14042 catalog_manager.cc:1558] T 00000000000000000000000000000000 P a0ea264bfe704cf1b386e5fefb6805d8: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 22:44:38.032284 14042 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 22:44:38.032356 14043 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 22:44:38.033346 14043 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 22:44:38.040562 14043 catalog_manager.cc:1347] Generated new cluster ID: 8cb9b786754543e28554fbbc4fbe723c
I20260925 22:44:38.040658 14043 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 22:44:38.042258 14024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:44:38.035965 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60192 (local address 127.10.198.62:36987)
0925 22:44:38.036132 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0925 22:44:38.036138 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 22:44:38.036389 (+ 251us) server_negotiation.cc:407] Connection header received
0925 22:44:38.036523 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:44:38.036526 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:44:38.036580 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:44:38.036695 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:44:38.037386 (+ 691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:44:38.038224 (+ 838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:44:38.039356 (+ 1132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:44:38.039755 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:44:38.039922 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:44:38.040940 (+ 1018us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:44:38.040943 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:44:38.040948 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 22:44:38.041576 (+ 628us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:44:38.041620 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 22:44:38.041646 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:44:38.041755 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0925 22:44:38.041961 (+ 206us) server_negotiation.cc:299] Negotiation successful
0925 22:44:38.042070 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260925 22:44:38.070497 14043 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 22:44:38.071671 14043 catalog_manager.cc:1504] Loading token signing keys...
I20260925 22:44:38.076522 14043 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a0ea264bfe704cf1b386e5fefb6805d8: Generated new TSK 0
I20260925 22:44:38.077478 14043 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 22:44:38.084197 11032 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrWXm8X/build/debug/bin/kudu with pid 13953
2026-09-25T22:44:38Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1467 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790376259650642-11032-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 22:44:38 dist-test-slave-5vdr krb5kdc[14054](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 22:44:38 dist-test-slave-5vdr krb5kdc[14054](info): set up 2 sockets
Sep 25 22:44:38 dist-test-slave-5vdr krb5kdc[14054](info): commencing operation
krb5kdc: starting...
W20260925 22:44:39.136053 11032 mini_kdc.cc:121] Time spent starting KDC: real 1.032s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790376259650642-11032-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790376259650642-11032-0/krb5kdc/test-user.keytab.
Sep 25 22:44:39 dist-test-slave-5vdr krb5kdc[14054](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376279, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T22:44:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T22:44:39Z Disabled control of system clock
WARNING: no policy specified for kudu/127.10.198.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.198.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.198.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790376259650642-11032-0/krb5kdc/kudu_127.10.198.62.keytab.
Entry for principal kudu/127.10.198.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790376259650642-11032-0/krb5kdc/kudu_127.10.198.62.keytab.
I20260925 22:44:39.268321 11032 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrWXm8X/build/debug/bin/kudu
/tmp/dist-test-taskrWXm8X/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790376259650642-11032-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790376259650642-11032-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790376259650642-11032-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790376259650642-11032-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.198.62:41215
--webserver_interface=127.10.198.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790376259650642-11032-0/krb5kdc/kudu_127.10.198.62.keytab
--principal=kudu/127.10.198.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.198.20:36185
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.198.62:41215
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790376259650642-11032-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790376259650642-11032-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790376259650642-11032-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:44:39.389529 14067 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:44:39.389851 14067 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:44:39.389895 14067 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:44:39.394367 14067 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 22:44:39.394454 14067 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:44:39.394474 14067 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:44:39.394491 14067 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 22:44:39.394507 14067 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 22:44:39.400487 14067 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.198.20:36185
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790376259650642-11032-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790376259650642-11032-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.198.62:41215
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790376259650642-11032-0/krb5kdc/kudu_127.10.198.62.keytab
--principal=kudu/127.10.198.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.198.62:41215
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790376259650642-11032-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.198.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14067
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790376259650642-11032-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:09:16 UTC on ccycloud.quasar-afmhyw.root.comops.site
build id 4673379
I20260925 22:44:39.401806 14067 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:44:39.402953 14067 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:44:39.412706 14073 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:44:39.412721 14072 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:44:39.412735 14075 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:44:39.413018 14067 server_base.cc:1034] running on GCE node
I20260925 22:44:39.413720 14067 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:44:39.415056 14067 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:44:39.416311 14067 hybrid_clock.cc:648] HybridClock initialized: now 1790376279416288 us; error 48 us; skew 500 ppm
Sep 25 22:44:39 dist-test-slave-5vdr krb5kdc[14054](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376279, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.198.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:44:39.419706 14067 init.cc:377] Logged in from keytab as kudu/127.10.198.62@KRBTEST.COM (short username kudu)
I20260925 22:44:39.420930 14067 webserver.cc:460] Webserver started at http://127.10.198.62:45879/ using document root <none> and password file <none>
I20260925 22:44:39.421604 14067 fs_manager.cc:359] Metadata directory not provided
I20260925 22:44:39.421725 14067 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:44:39.422046 14067 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:44:39.424463 14067 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790376259650642-11032-0/minicluster-data/master-0/data/instance:
uuid: "05fa7b7797574fa48653feb01a25813a"
format_stamp: "Formatted at 2026-09-25 22:44:39 on dist-test-slave-5vdr"
server_key: "a5d2ab51bcd38c2a2a5f67dcde99c29e"
server_key_iv: "ebdcc23d09af2de94c799efaf6e59d28"
server_key_version: "encryptionkey@0"
I20260925 22:44:39.425133 14067 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790376259650642-11032-0/minicluster-data/master-0/wal/instance:
uuid: "05fa7b7797574fa48653feb01a25813a"
format_stamp: "Formatted at 2026-09-25 22:44:39 on dist-test-slave-5vdr"
server_key: "a5d2ab51bcd38c2a2a5f67dcde99c29e"
server_key_iv: "ebdcc23d09af2de94c799efaf6e59d28"
server_key_version: "encryptionkey@0"
I20260925 22:44:39.429471 14067 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260925 22:44:39.432518 14082 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:44:39.434337 14067 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 22:44:39.434485 14067 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790376259650642-11032-0/minicluster-data/master-0/data,/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790376259650642-11032-0/minicluster-data/master-0/wal
uuid: "05fa7b7797574fa48653feb01a25813a"
format_stamp: "Formatted at 2026-09-25 22:44:39 on dist-test-slave-5vdr"
server_key: "a5d2ab51bcd38c2a2a5f67dcde99c29e"
server_key_iv: "ebdcc23d09af2de94c799efaf6e59d28"
server_key_version: "encryptionkey@0"
I20260925 22:44:39.434613 14067 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790376259650642-11032-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790376259650642-11032-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790376259650642-11032-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:44:39.453612 14067 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:44:39.457417 14067 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:44:39.457610 14067 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:44:39.465220 14067 rpc_server.cc:307] RPC server started. Bound to: 127.10.198.62:41215
I20260925 22:44:39.465297 14134 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.198.62:41215 every 8 connection(s)
I20260925 22:44:39.466368 14067 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790376259650642-11032-0/minicluster-data/master-0/data/info.pb
I20260925 22:44:39.469897 14135 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:44:39.476105 11032 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrWXm8X/build/debug/bin/kudu as pid 14067
I20260925 22:44:39.476226 11032 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790376259650642-11032-0/minicluster-data/master-0/wal/instance
I20260925 22:44:39.476493 11032 external_mini_cluster.cc:1428] Setting key 8ff8817b96f9a60000754df6f4b3e8b4
I20260925 22:44:39.477401 14135 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 05fa7b7797574fa48653feb01a25813a: Bootstrap starting.
I20260925 22:44:39.480425 14135 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 05fa7b7797574fa48653feb01a25813a: Neither blocks nor log segments found. Creating new log.
I20260925 22:44:39.481384 14135 log.cc:824] T 00000000000000000000000000000000 P 05fa7b7797574fa48653feb01a25813a: Log is configured to *not* fsync() on all Append() calls
I20260925 22:44:39.484014 14135 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 05fa7b7797574fa48653feb01a25813a: No bootstrap required, opened a new log
Sep 25 22:44:39 dist-test-slave-5vdr krb5kdc[14054](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376279, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.198.62@KRBTEST.COM
I20260925 22:44:39.488811 14135 raft_consensus.cc:348] T 00000000000000000000000000000000 P 05fa7b7797574fa48653feb01a25813a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05fa7b7797574fa48653feb01a25813a" member_type: VOTER last_known_addr { host: "127.10.198.62" port: 41215 } }
I20260925 22:44:39.489167 14135 raft_consensus.cc:374] T 00000000000000000000000000000000 P 05fa7b7797574fa48653feb01a25813a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:44:39.489249 14135 raft_consensus.cc:729] T 00000000000000000000000000000000 P 05fa7b7797574fa48653feb01a25813a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 05fa7b7797574fa48653feb01a25813a, State: Initialized, Role: FOLLOWER
I20260925 22:44:39.489907 14135 consensus_queue.cc:260] T 00000000000000000000000000000000 P 05fa7b7797574fa48653feb01a25813a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05fa7b7797574fa48653feb01a25813a" member_type: VOTER last_known_addr { host: "127.10.198.62" port: 41215 } }
I20260925 22:44:39.490164 14135 raft_consensus.cc:388] T 00000000000000000000000000000000 P 05fa7b7797574fa48653feb01a25813a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 22:44:39.490345 14135 raft_consensus.cc:482] T 00000000000000000000000000000000 P 05fa7b7797574fa48653feb01a25813a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 22:44:39.490540 14135 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 05fa7b7797574fa48653feb01a25813a [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:44:39.491926 14135 raft_consensus.cc:504] T 00000000000000000000000000000000 P 05fa7b7797574fa48653feb01a25813a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05fa7b7797574fa48653feb01a25813a" member_type: VOTER last_known_addr { host: "127.10.198.62" port: 41215 } }
I20260925 22:44:39.492460 14135 leader_election.cc:302] T 00000000000000000000000000000000 P 05fa7b7797574fa48653feb01a25813a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 05fa7b7797574fa48653feb01a25813a; no voters:
I20260925 22:44:39.492882 14135 leader_election.cc:288] T 00000000000000000000000000000000 P 05fa7b7797574fa48653feb01a25813a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 22:44:39.493021 14140 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 05fa7b7797574fa48653feb01a25813a [term 1 FOLLOWER]: Leader election won for term 1
I20260925 22:44:39.493271 14140 raft_consensus.cc:686] T 00000000000000000000000000000000 P 05fa7b7797574fa48653feb01a25813a [term 1 LEADER]: Becoming Leader. State: Replica: 05fa7b7797574fa48653feb01a25813a, State: Running, Role: LEADER
I20260925 22:44:39.493691 14140 consensus_queue.cc:237] T 00000000000000000000000000000000 P 05fa7b7797574fa48653feb01a25813a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05fa7b7797574fa48653feb01a25813a" member_type: VOTER last_known_addr { host: "127.10.198.62" port: 41215 } }
I20260925 22:44:39.493906 14138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:44:39.478150 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39218 (local address 127.10.198.62:41215)
0925 22:44:39.478707 (+ 557us) server_negotiation.cc:206] Beginning negotiation
0925 22:44:39.478724 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0925 22:44:39.478764 (+ 40us) server_negotiation.cc:407] Connection header received
0925 22:44:39.479502 (+ 738us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:44:39.479524 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:44:39.480014 (+ 490us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:44:39.480428 (+ 414us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:44:39.481322 (+ 894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:44:39.482455 (+ 1133us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:44:39.483404 (+ 949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:44:39.484020 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:44:39.484203 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:44:39.487303 (+ 3100us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:44:39.487310 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:44:39.487329 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:44:39.487370 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:44:39.489526 (+ 2156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:44:39.490305 (+ 779us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:44:39.490312 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:44:39.490320 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:44:39.490522 (+ 202us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:44:39.490958 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:44:39.490961 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:44:39.490963 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:44:39.491465 (+ 502us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:44:39.491633 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0925 22:44:39.492185 (+ 552us) server_negotiation.cc:299] Negotiation successful
0925 22:44:39.492627 (+ 442us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":374,"thread_start_us":177,"threads_started":1}
I20260925 22:44:39.494483 14135 sys_catalog.cc:564] T 00000000000000000000000000000000 P 05fa7b7797574fa48653feb01a25813a [sys.catalog]: configured and running, proceeding with master startup.
I20260925 22:44:39.495079 14140 sys_catalog.cc:455] T 00000000000000000000000000000000 P 05fa7b7797574fa48653feb01a25813a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 05fa7b7797574fa48653feb01a25813a. Latest consensus state: current_term: 1 leader_uuid: "05fa7b7797574fa48653feb01a25813a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05fa7b7797574fa48653feb01a25813a" member_type: VOTER last_known_addr { host: "127.10.198.62" port: 41215 } } }
I20260925 22:44:39.495965 14140 sys_catalog.cc:458] T 00000000000000000000000000000000 P 05fa7b7797574fa48653feb01a25813a [sys.catalog]: This master's current role is: LEADER
I20260925 22:44:39.495793 14141 sys_catalog.cc:455] T 00000000000000000000000000000000 P 05fa7b7797574fa48653feb01a25813a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "05fa7b7797574fa48653feb01a25813a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05fa7b7797574fa48653feb01a25813a" member_type: VOTER last_known_addr { host: "127.10.198.62" port: 41215 } } }
I20260925 22:44:39.496265 14141 sys_catalog.cc:458] T 00000000000000000000000000000000 P 05fa7b7797574fa48653feb01a25813a [sys.catalog]: This master's current role is: LEADER
I20260925 22:44:39.496628 14149 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 22:44:39.500326 14149 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 22:44:39.507830 14149 catalog_manager.cc:1347] Generated new cluster ID: 1fa86f752fb749ffaa3e02a436d7998a
I20260925 22:44:39.508033 14149 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 22:44:39.531922 14149 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 22:44:39.533005 14149 catalog_manager.cc:1504] Loading token signing keys...
I20260925 22:44:39.541416 14149 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 05fa7b7797574fa48653feb01a25813a: Generated new TSK 0
I20260925 22:44:39.542477 14149 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.10.198.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.198.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.198.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790376259650642-11032-0/krb5kdc/kudu_127.10.198.1.keytab.
Entry for principal kudu/127.10.198.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790376259650642-11032-0/krb5kdc/kudu_127.10.198.1.keytab.
I20260925 22:44:39.579221 11032 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrWXm8X/build/debug/bin/kudu
/tmp/dist-test-taskrWXm8X/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790376259650642-11032-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790376259650642-11032-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790376259650642-11032-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790376259650642-11032-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.198.1:0
--local_ip_for_outbound_sockets=127.10.198.1
--webserver_interface=127.10.198.1
--webserver_port=0
--tserver_master_addrs=127.10.198.62:41215
--keytab_file=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790376259650642-11032-0/krb5kdc/kudu_127.10.198.1.keytab
--principal=kudu/127.10.198.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.198.20:36185
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790376259650642-11032-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790376259650642-11032-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790376259650642-11032-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:44:39.701087 14161 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:44:39.701413 14161 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:44:39.701464 14161 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:44:39.705991 14161 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:44:39.706073 14161 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:44:39.706137 14161 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.198.1
I20260925 22:44:39.712577 14161 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.198.20:36185
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790376259650642-11032-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790376259650642-11032-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790376259650642-11032-0/krb5kdc/kudu_127.10.198.1.keytab
--principal=kudu/127.10.198.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.198.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790376259650642-11032-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.198.1
--webserver_port=0
--tserver_master_addrs=127.10.198.62:41215
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14161
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.198.1
--log_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790376259650642-11032-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:09:16 UTC on ccycloud.quasar-afmhyw.root.comops.site
build id 4673379
I20260925 22:44:39.713797 14161 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:44:39.714958 14161 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:44:39.725647 14166 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:44:39.725647 14167 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:44:39.725678 14169 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:44:39.726357 14161 server_base.cc:1034] running on GCE node
I20260925 22:44:39.726946 14161 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:44:39.727824 14161 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:44:39.729123 14161 hybrid_clock.cc:648] HybridClock initialized: now 1790376279729105 us; error 37 us; skew 500 ppm
Sep 25 22:44:39 dist-test-slave-5vdr krb5kdc[14054](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376279, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.198.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:44:39.732506 14161 init.cc:377] Logged in from keytab as kudu/127.10.198.1@KRBTEST.COM (short username kudu)
I20260925 22:44:39.733554 14161 webserver.cc:460] Webserver started at http://127.10.198.1:39057/ using document root <none> and password file <none>
I20260925 22:44:39.734149 14161 fs_manager.cc:359] Metadata directory not provided
I20260925 22:44:39.734225 14161 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:44:39.734418 14161 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:44:39.736784 14161 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790376259650642-11032-0/minicluster-data/ts-0/data/instance:
uuid: "6fd4cbcd6cda404b92fada566c512589"
format_stamp: "Formatted at 2026-09-25 22:44:39 on dist-test-slave-5vdr"
server_key: "8052dbfba9e58f9f0e2aad8fcb46de96"
server_key_iv: "d2295d455b74539b8dc9b58d35b078f9"
server_key_version: "encryptionkey@0"
I20260925 22:44:39.737756 14161 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790376259650642-11032-0/minicluster-data/ts-0/wal/instance:
uuid: "6fd4cbcd6cda404b92fada566c512589"
format_stamp: "Formatted at 2026-09-25 22:44:39 on dist-test-slave-5vdr"
server_key: "8052dbfba9e58f9f0e2aad8fcb46de96"
server_key_iv: "d2295d455b74539b8dc9b58d35b078f9"
server_key_version: "encryptionkey@0"
I20260925 22:44:39.743382 14161 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.004s
I20260925 22:44:39.746981 14176 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:44:39.749114 14161 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260925 22:44:39.749298 14161 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790376259650642-11032-0/minicluster-data/ts-0/data,/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790376259650642-11032-0/minicluster-data/ts-0/wal
uuid: "6fd4cbcd6cda404b92fada566c512589"
format_stamp: "Formatted at 2026-09-25 22:44:39 on dist-test-slave-5vdr"
server_key: "8052dbfba9e58f9f0e2aad8fcb46de96"
server_key_iv: "d2295d455b74539b8dc9b58d35b078f9"
server_key_version: "encryptionkey@0"
I20260925 22:44:39.749452 14161 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790376259650642-11032-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790376259650642-11032-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790376259650642-11032-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:44:39.760530 14161 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:44:39.764204 14161 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:44:39.764451 14161 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:44:39.765113 14161 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 22:44:39.766032 14161 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:44:39.766085 14161 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:44:39.766129 14161 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:44:39.766181 14161 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 22:44:39.775909 14161 rpc_server.cc:307] RPC server started. Bound to: 127.10.198.1:34575
I20260925 22:44:39.775925 14289 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.198.1:34575 every 8 connection(s)
I20260925 22:44:39.776968 14161 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790376259650642-11032-0/minicluster-data/ts-0/data/info.pb
I20260925 22:44:39.787171 11032 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrWXm8X/build/debug/bin/kudu as pid 14161
I20260925 22:44:39.787379 11032 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790376259650642-11032-0/minicluster-data/ts-0/wal/instance
I20260925 22:44:39.787797 11032 external_mini_cluster.cc:1428] Setting key aa78f1d183cfa5b5240087a5e16cf4bc
Sep 25 22:44:39 dist-test-slave-5vdr krb5kdc[14054](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376279, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.198.1@KRBTEST.COM for kudu/127.10.198.62@KRBTEST.COM
I20260925 22:44:39.793154 14138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:44:39.779053 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.198.1:44429 (local address 127.10.198.62:41215)
0925 22:44:39.779213 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0925 22:44:39.779217 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:44:39.780237 (+ 1020us) server_negotiation.cc:407] Connection header received
0925 22:44:39.781717 (+ 1480us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:44:39.781721 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:44:39.781786 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:44:39.781904 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:44:39.783260 (+ 1356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:44:39.783817 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:44:39.785154 (+ 1337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:44:39.785399 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:44:39.785530 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:44:39.788831 (+ 3301us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:44:39.788834 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:44:39.788836 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:44:39.788865 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:44:39.790721 (+ 1856us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:44:39.791364 (+ 643us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:44:39.791368 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:44:39.791370 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:44:39.791466 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:44:39.791905 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:44:39.791909 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:44:39.791911 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:44:39.792122 (+ 211us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:44:39.792245 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0925 22:44:39.792830 (+ 585us) server_negotiation.cc:299] Negotiation successful
0925 22:44:39.792979 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260925 22:44:39.794097 14292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:44:39.779489 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.198.62:41215 (local address 127.10.198.1:44429)
0925 22:44:39.780062 (+ 573us) negotiation.cc:107] Waiting for socket to connect
0925 22:44:39.780111 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0925 22:44:39.781436 (+ 1325us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:44:39.782121 (+ 685us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:44:39.782129 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:44:39.782537 (+ 408us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:44:39.783072 (+ 535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:44:39.783085 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:44:39.784030 (+ 945us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:44:39.784036 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:44:39.785001 (+ 965us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:44:39.785012 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:44:39.785580 (+ 568us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:44:39.785584 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:44:39.785775 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:44:39.786464 (+ 689us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:44:39.786481 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:44:39.788619 (+ 2138us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:44:39.790884 (+ 2265us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:44:39.790893 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:44:39.790913 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:44:39.791235 (+ 322us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:44:39.791572 (+ 337us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:44:39.791575 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:44:39.791577 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:44:39.791764 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:44:39.792243 (+ 479us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:44:39.792253 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:44:39.792561 (+ 308us) client_negotiation.cc:769] Sending connection context
0925 22:44:39.792776 (+ 215us) client_negotiation.cc:240] Negotiation successful
0925 22:44:39.793149 (+ 373us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":297,"thread_start_us":122,"threads_started":1}
I20260925 22:44:39.795729 14290 heartbeater.cc:347] Connected to a master server at 127.10.198.62:41215
I20260925 22:44:39.796069 14290 heartbeater.cc:464] Registering TS with master...
I20260925 22:44:39.796645 14290 heartbeater.cc:511] Master 127.10.198.62:41215 requested a full tablet report, sending...
I20260925 22:44:39.798559 14097 ts_manager.cc:194] Registered new tserver with Master: 6fd4cbcd6cda404b92fada566c512589 (127.10.198.1:34575)
WARNING: no policy specified for kudu/127.10.198.2@KRBTEST.COM; defaulting to no policy
I20260925 22:44:39.799942 14097 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.198.1@KRBTEST.COM'} at 127.10.198.1:44429
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.198.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.198.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790376259650642-11032-0/krb5kdc/kudu_127.10.198.2.keytab.
Entry for principal kudu/127.10.198.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790376259650642-11032-0/krb5kdc/kudu_127.10.198.2.keytab.
I20260925 22:44:39.820375 11032 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrWXm8X/build/debug/bin/kudu
/tmp/dist-test-taskrWXm8X/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790376259650642-11032-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790376259650642-11032-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790376259650642-11032-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790376259650642-11032-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.198.2:0
--local_ip_for_outbound_sockets=127.10.198.2
--webserver_interface=127.10.198.2
--webserver_port=0
--tserver_master_addrs=127.10.198.62:41215
--keytab_file=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790376259650642-11032-0/krb5kdc/kudu_127.10.198.2.keytab
--principal=kudu/127.10.198.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.198.20:36185
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790376259650642-11032-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790376259650642-11032-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790376259650642-11032-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:44:39.940249 14295 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:44:39.940567 14295 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:44:39.940613 14295 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:44:39.945042 14295 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:44:39.945139 14295 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:44:39.945205 14295 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.198.2
I20260925 22:44:39.951086 14295 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.198.20:36185
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790376259650642-11032-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790376259650642-11032-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790376259650642-11032-0/krb5kdc/kudu_127.10.198.2.keytab
--principal=kudu/127.10.198.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.198.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790376259650642-11032-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.198.2
--webserver_port=0
--tserver_master_addrs=127.10.198.62:41215
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14295
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.198.2
--log_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790376259650642-11032-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:09:16 UTC on ccycloud.quasar-afmhyw.root.comops.site
build id 4673379
I20260925 22:44:39.952976 14295 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:44:39.954653 14295 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:44:39.965551 14300 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:44:39.965567 14303 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:44:39.965567 14301 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:44:39.965658 14295 server_base.cc:1034] running on GCE node
I20260925 22:44:39.966325 14295 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:44:39.967057 14295 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:44:39.968255 14295 hybrid_clock.cc:648] HybridClock initialized: now 1790376279968209 us; error 66 us; skew 500 ppm
Sep 25 22:44:39 dist-test-slave-5vdr krb5kdc[14054](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376279, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.198.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:44:39.971597 14295 init.cc:377] Logged in from keytab as kudu/127.10.198.2@KRBTEST.COM (short username kudu)
I20260925 22:44:39.972755 14295 webserver.cc:460] Webserver started at http://127.10.198.2:37707/ using document root <none> and password file <none>
I20260925 22:44:39.973439 14295 fs_manager.cc:359] Metadata directory not provided
I20260925 22:44:39.973513 14295 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:44:39.973785 14295 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:44:39.976173 14295 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790376259650642-11032-0/minicluster-data/ts-1/data/instance:
uuid: "791c6a8b98064b2c91f239261e6558d0"
format_stamp: "Formatted at 2026-09-25 22:44:39 on dist-test-slave-5vdr"
server_key: "65938cfc9385a9e3e648e36261f46054"
server_key_iv: "a06570e3953c1195f7c99a7567e8562f"
server_key_version: "encryptionkey@0"
I20260925 22:44:39.976821 14295 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790376259650642-11032-0/minicluster-data/ts-1/wal/instance:
uuid: "791c6a8b98064b2c91f239261e6558d0"
format_stamp: "Formatted at 2026-09-25 22:44:39 on dist-test-slave-5vdr"
server_key: "65938cfc9385a9e3e648e36261f46054"
server_key_iv: "a06570e3953c1195f7c99a7567e8562f"
server_key_version: "encryptionkey@0"
I20260925 22:44:39.981673 14295 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260925 22:44:39.984680 14310 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:44:39.986176 14295 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 22:44:39.986364 14295 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790376259650642-11032-0/minicluster-data/ts-1/data,/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790376259650642-11032-0/minicluster-data/ts-1/wal
uuid: "791c6a8b98064b2c91f239261e6558d0"
format_stamp: "Formatted at 2026-09-25 22:44:39 on dist-test-slave-5vdr"
server_key: "65938cfc9385a9e3e648e36261f46054"
server_key_iv: "a06570e3953c1195f7c99a7567e8562f"
server_key_version: "encryptionkey@0"
I20260925 22:44:39.986505 14295 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790376259650642-11032-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790376259650642-11032-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790376259650642-11032-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:44:40.004734 14295 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:44:40.008497 14295 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:44:40.008734 14295 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:44:40.009403 14295 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 22:44:40.010329 14295 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:44:40.010404 14295 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:44:40.010480 14295 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:44:40.010517 14295 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 22:44:40.020306 14295 rpc_server.cc:307] RPC server started. Bound to: 127.10.198.2:34477
I20260925 22:44:40.020324 14423 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.198.2:34477 every 8 connection(s)
I20260925 22:44:40.021344 14295 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790376259650642-11032-0/minicluster-data/ts-1/data/info.pb
I20260925 22:44:40.028172 11032 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrWXm8X/build/debug/bin/kudu as pid 14295
I20260925 22:44:40.028321 11032 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790376259650642-11032-0/minicluster-data/ts-1/wal/instance
I20260925 22:44:40.028641 11032 external_mini_cluster.cc:1428] Setting key 4fb9a6d6b9af83c9cc62c9484bde4a7e
Sep 25 22:44:40 dist-test-slave-5vdr krb5kdc[14054](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376279, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.198.2@KRBTEST.COM for kudu/127.10.198.62@KRBTEST.COM
I20260925 22:44:40.037334 14138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:44:40.023531 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.198.2:42113 (local address 127.10.198.62:41215)
0925 22:44:40.023705 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0925 22:44:40.023711 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 22:44:40.024730 (+ 1019us) server_negotiation.cc:407] Connection header received
0925 22:44:40.026159 (+ 1429us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:44:40.026163 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:44:40.026215 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:44:40.026296 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:44:40.027636 (+ 1340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:44:40.028268 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:44:40.029949 (+ 1681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:44:40.030294 (+ 345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:44:40.030413 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:44:40.033278 (+ 2865us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:44:40.033281 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:44:40.033283 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:44:40.033312 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:44:40.034941 (+ 1629us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:44:40.035565 (+ 624us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:44:40.035570 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:44:40.035574 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:44:40.035661 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:44:40.036125 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:44:40.036129 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:44:40.036130 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:44:40.036336 (+ 206us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:44:40.036453 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0925 22:44:40.037021 (+ 568us) server_negotiation.cc:299] Negotiation successful
0925 22:44:40.037160 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
WARNING: no policy specified for kudu/127.10.198.3@KRBTEST.COM; defaulting to no policy
I20260925 22:44:40.038370 14426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:44:40.024047 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.198.62:41215 (local address 127.10.198.2:42113)
0925 22:44:40.024546 (+ 499us) negotiation.cc:107] Waiting for socket to connect
0925 22:44:40.024597 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0925 22:44:40.025903 (+ 1306us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:44:40.026503 (+ 600us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:44:40.026511 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:44:40.026922 (+ 411us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:44:40.027469 (+ 547us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:44:40.027483 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:44:40.028490 (+ 1007us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:44:40.028499 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:44:40.029766 (+ 1267us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:44:40.029780 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:44:40.030503 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:44:40.030508 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:44:40.030718 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:44:40.031400 (+ 682us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:44:40.031419 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:44:40.033042 (+ 1623us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:44:40.035098 (+ 2056us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:44:40.035105 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:44:40.035127 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:44:40.035444 (+ 317us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:44:40.035767 (+ 323us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:44:40.035770 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:44:40.035772 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:44:40.036018 (+ 246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:44:40.036460 (+ 442us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:44:40.036470 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:44:40.036758 (+ 288us) client_negotiation.cc:769] Sending connection context
0925 22:44:40.036958 (+ 200us) client_negotiation.cc:240] Negotiation successful
0925 22:44:40.037323 (+ 365us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":286,"thread_start_us":115,"threads_started":1}
I20260925 22:44:40.039803 14424 heartbeater.cc:347] Connected to a master server at 127.10.198.62:41215
I20260925 22:44:40.040174 14424 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.198.3@KRBTEST.COM" created.
I20260925 22:44:40.040904 14424 heartbeater.cc:511] Master 127.10.198.62:41215 requested a full tablet report, sending...
I20260925 22:44:40.042192 14097 ts_manager.cc:194] Registered new tserver with Master: 791c6a8b98064b2c91f239261e6558d0 (127.10.198.2:34477)
I20260925 22:44:40.043109 14097 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.198.2@KRBTEST.COM'} at 127.10.198.2:42113
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.198.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790376259650642-11032-0/krb5kdc/kudu_127.10.198.3.keytab.
Entry for principal kudu/127.10.198.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790376259650642-11032-0/krb5kdc/kudu_127.10.198.3.keytab.
I20260925 22:44:40.059183 11032 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrWXm8X/build/debug/bin/kudu
/tmp/dist-test-taskrWXm8X/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790376259650642-11032-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790376259650642-11032-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790376259650642-11032-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790376259650642-11032-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.198.3:0
--local_ip_for_outbound_sockets=127.10.198.3
--webserver_interface=127.10.198.3
--webserver_port=0
--tserver_master_addrs=127.10.198.62:41215
--keytab_file=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790376259650642-11032-0/krb5kdc/kudu_127.10.198.3.keytab
--principal=kudu/127.10.198.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.198.20:36185
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790376259650642-11032-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790376259650642-11032-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790376259650642-11032-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:44:40.179075 14429 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:44:40.179456 14429 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:44:40.179534 14429 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:44:40.184015 14429 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:44:40.184122 14429 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:44:40.184221 14429 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.198.3
I20260925 22:44:40.190794 14429 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.198.20:36185
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790376259650642-11032-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790376259650642-11032-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790376259650642-11032-0/krb5kdc/kudu_127.10.198.3.keytab
--principal=kudu/127.10.198.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.198.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790376259650642-11032-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.198.3
--webserver_port=0
--tserver_master_addrs=127.10.198.62:41215
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14429
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.198.3
--log_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790376259650642-11032-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:09:16 UTC on ccycloud.quasar-afmhyw.root.comops.site
build id 4673379
I20260925 22:44:40.192268 14429 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:44:40.193463 14429 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:44:40.204650 14437 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:44:40.204649 14435 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:44:40.204653 14434 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:44:40.205171 14429 server_base.cc:1034] running on GCE node
I20260925 22:44:40.205736 14429 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:44:40.206584 14429 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:44:40.207828 14429 hybrid_clock.cc:648] HybridClock initialized: now 1790376280207787 us; error 69 us; skew 500 ppm
Sep 25 22:44:40 dist-test-slave-5vdr krb5kdc[14054](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376280, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.198.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:44:40.211405 14429 init.cc:377] Logged in from keytab as kudu/127.10.198.3@KRBTEST.COM (short username kudu)
I20260925 22:44:40.212991 14429 webserver.cc:460] Webserver started at http://127.10.198.3:46017/ using document root <none> and password file <none>
I20260925 22:44:40.213685 14429 fs_manager.cc:359] Metadata directory not provided
I20260925 22:44:40.213814 14429 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:44:40.214074 14429 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:44:40.216478 14429 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790376259650642-11032-0/minicluster-data/ts-2/data/instance:
uuid: "40d057a7d630473e886ec1d662d0add5"
format_stamp: "Formatted at 2026-09-25 22:44:40 on dist-test-slave-5vdr"
server_key: "66083d40712c98fdb529713822b79197"
server_key_iv: "7e208752c2e5c2fb93b6b45eab53dd6e"
server_key_version: "encryptionkey@0"
I20260925 22:44:40.217182 14429 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790376259650642-11032-0/minicluster-data/ts-2/wal/instance:
uuid: "40d057a7d630473e886ec1d662d0add5"
format_stamp: "Formatted at 2026-09-25 22:44:40 on dist-test-slave-5vdr"
server_key: "66083d40712c98fdb529713822b79197"
server_key_iv: "7e208752c2e5c2fb93b6b45eab53dd6e"
server_key_version: "encryptionkey@0"
I20260925 22:44:40.221660 14429 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260925 22:44:40.224774 14444 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:44:40.226207 14429 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 22:44:40.226399 14429 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790376259650642-11032-0/minicluster-data/ts-2/data,/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790376259650642-11032-0/minicluster-data/ts-2/wal
uuid: "40d057a7d630473e886ec1d662d0add5"
format_stamp: "Formatted at 2026-09-25 22:44:40 on dist-test-slave-5vdr"
server_key: "66083d40712c98fdb529713822b79197"
server_key_iv: "7e208752c2e5c2fb93b6b45eab53dd6e"
server_key_version: "encryptionkey@0"
I20260925 22:44:40.226542 14429 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790376259650642-11032-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790376259650642-11032-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790376259650642-11032-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:44:40.248785 14429 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:44:40.252085 14429 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:44:40.252314 14429 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:44:40.253022 14429 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 22:44:40.253944 14429 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:44:40.254029 14429 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:44:40.254101 14429 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:44:40.254153 14429 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 22:44:40.264117 14429 rpc_server.cc:307] RPC server started. Bound to: 127.10.198.3:45153
I20260925 22:44:40.264132 14557 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.198.3:45153 every 8 connection(s)
I20260925 22:44:40.265184 14429 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790376259650642-11032-0/minicluster-data/ts-2/data/info.pb
I20260925 22:44:40.266808 11032 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrWXm8X/build/debug/bin/kudu as pid 14429
I20260925 22:44:40.266970 11032 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790376259650642-11032-0/minicluster-data/ts-2/wal/instance
I20260925 22:44:40.267349 11032 external_mini_cluster.cc:1428] Setting key 4c22176a5b06b2d79f035b12089dbbbd
Sep 25 22:44:40 dist-test-slave-5vdr krb5kdc[14054](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376280, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.198.3@KRBTEST.COM for kudu/127.10.198.62@KRBTEST.COM
I20260925 22:44:40.281076 14138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:44:40.267285 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.198.3:33135 (local address 127.10.198.62:41215)
0925 22:44:40.267462 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0925 22:44:40.267467 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:44:40.268659 (+ 1192us) server_negotiation.cc:407] Connection header received
0925 22:44:40.270495 (+ 1836us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:44:40.270498 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:44:40.270562 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:44:40.270643 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:44:40.272003 (+ 1360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:44:40.272552 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:44:40.273812 (+ 1260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:44:40.274052 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:44:40.274180 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:44:40.277082 (+ 2902us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:44:40.277085 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:44:40.277087 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:44:40.277117 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:44:40.278481 (+ 1364us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:44:40.279143 (+ 662us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:44:40.279146 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:44:40.279148 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:44:40.279264 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:44:40.279690 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:44:40.279692 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:44:40.279694 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:44:40.280001 (+ 307us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:44:40.280133 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0925 22:44:40.280713 (+ 580us) server_negotiation.cc:299] Negotiation successful
0925 22:44:40.280869 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260925 22:44:40.282168 14560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:44:40.267790 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.198.62:41215 (local address 127.10.198.3:33135)
0925 22:44:40.268499 (+ 709us) negotiation.cc:107] Waiting for socket to connect
0925 22:44:40.268561 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0925 22:44:40.270260 (+ 1699us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:44:40.270851 (+ 591us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:44:40.270860 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:44:40.271260 (+ 400us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:44:40.271781 (+ 521us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:44:40.271795 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:44:40.272700 (+ 905us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:44:40.272704 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:44:40.273661 (+ 957us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:44:40.273671 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:44:40.274229 (+ 558us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:44:40.274233 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:44:40.274443 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:44:40.275120 (+ 677us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:44:40.275139 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:44:40.276853 (+ 1714us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:44:40.278619 (+ 1766us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:44:40.278626 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:44:40.278646 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:44:40.278974 (+ 328us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:44:40.279396 (+ 422us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:44:40.279399 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:44:40.279400 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:44:40.279579 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:44:40.280129 (+ 550us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:44:40.280136 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:44:40.280428 (+ 292us) client_negotiation.cc:769] Sending connection context
0925 22:44:40.280627 (+ 199us) client_negotiation.cc:240] Negotiation successful
0925 22:44:40.281068 (+ 441us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":436,"thread_start_us":169,"threads_started":1}
I20260925 22:44:40.283490 14558 heartbeater.cc:347] Connected to a master server at 127.10.198.62:41215
I20260925 22:44:40.283923 14558 heartbeater.cc:464] Registering TS with master...
I20260925 22:44:40.284519 14558 heartbeater.cc:511] Master 127.10.198.62:41215 requested a full tablet report, sending...
I20260925 22:44:40.285821 14097 ts_manager.cc:194] Registered new tserver with Master: 40d057a7d630473e886ec1d662d0add5 (127.10.198.3:45153)
I20260925 22:44:40.286449 14097 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.198.3@KRBTEST.COM'} at 127.10.198.3:33135
I20260925 22:44:40.294421 11032 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 25 22:44:40 dist-test-slave-5vdr krb5kdc[14054](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376280, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Sep 25 22:44:40 dist-test-slave-5vdr krb5kdc[14054](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376280, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.10.198.62@KRBTEST.COM
I20260925 22:44:40.321978 14138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:44:40.313111 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39224 (local address 127.10.198.62:41215)
0925 22:44:40.313341 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0925 22:44:40.313345 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:44:40.313361 (+ 16us) server_negotiation.cc:407] Connection header received
0925 22:44:40.313523 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:44:40.313525 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:44:40.313572 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:44:40.313645 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:44:40.314381 (+ 736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:44:40.314908 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:44:40.315798 (+ 890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:44:40.316097 (+ 299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:44:40.316186 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:44:40.318773 (+ 2587us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:44:40.318777 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:44:40.318779 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:44:40.318813 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:44:40.320298 (+ 1485us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:44:40.320770 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:44:40.320772 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:44:40.320774 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:44:40.320857 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:44:40.321100 (+ 243us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:44:40.321102 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:44:40.321103 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:44:40.321334 (+ 231us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:44:40.321444 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0925 22:44:40.321652 (+ 208us) server_negotiation.cc:299] Negotiation successful
0925 22:44:40.321759 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":100}
W20260925 22:44:40.322502 14097 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:39224
I20260925 22:44:40.323510 11032 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrWXm8X/build/debug/bin/kudu with pid 14161
I20260925 22:44:40.329190 11032 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrWXm8X/build/debug/bin/kudu with pid 14295
I20260925 22:44:40.334893 11032 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrWXm8X/build/debug/bin/kudu with pid 14429
I20260925 22:44:40.340957 11032 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrWXm8X/build/debug/bin/kudu with pid 14067
2026-09-25T22:44:40Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (2259 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790376259650642-11032-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 22:44:40 dist-test-slave-5vdr krb5kdc[14570](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 22:44:40 dist-test-slave-5vdr krb5kdc[14570](info): set up 2 sockets
Sep 25 22:44:40 dist-test-slave-5vdr krb5kdc[14570](info): commencing operation
krb5kdc: starting...
W20260925 22:44:41.388123 11032 mini_kdc.cc:121] Time spent starting KDC: real 1.024s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790376259650642-11032-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790376259650642-11032-0/krb5kdc/test-user.keytab.
Sep 25 22:44:41 dist-test-slave-5vdr krb5kdc[14570](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376281, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T22:44:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T22:44:41Z Disabled control of system clock
WARNING: no policy specified for kudu/127.10.198.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.198.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.198.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790376259650642-11032-0/krb5kdc/kudu_127.10.198.62.keytab.
Entry for principal kudu/127.10.198.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790376259650642-11032-0/krb5kdc/kudu_127.10.198.62.keytab.
I20260925 22:44:41.525609 11032 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrWXm8X/build/debug/bin/kudu
/tmp/dist-test-taskrWXm8X/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790376259650642-11032-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790376259650642-11032-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790376259650642-11032-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790376259650642-11032-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.198.62:41515
--webserver_interface=127.10.198.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790376259650642-11032-0/krb5kdc/kudu_127.10.198.62.keytab
--principal=kudu/127.10.198.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.198.20:38063
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.198.62:41515
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790376259650642-11032-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790376259650642-11032-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790376259650642-11032-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:44:41.653833 14583 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:44:41.654220 14583 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:44:41.654302 14583 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:44:41.658653 14583 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 22:44:41.658792 14583 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:44:41.658829 14583 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:44:41.658871 14583 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 22:44:41.658908 14583 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 22:44:41.664914 14583 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.198.20:38063
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790376259650642-11032-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790376259650642-11032-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.198.62:41515
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790376259650642-11032-0/krb5kdc/kudu_127.10.198.62.keytab
--principal=kudu/127.10.198.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.198.62:41515
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790376259650642-11032-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.198.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14583
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790376259650642-11032-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:09:16 UTC on ccycloud.quasar-afmhyw.root.comops.site
build id 4673379
I20260925 22:44:41.666181 14583 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:44:41.667325 14583 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:44:41.676853 14588 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:44:41.676856 14591 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:44:41.676929 14589 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:44:41.677251 14583 server_base.cc:1034] running on GCE node
I20260925 22:44:41.678165 14583 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:44:41.679963 14583 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:44:41.681195 14583 hybrid_clock.cc:648] HybridClock initialized: now 1790376281681158 us; error 64 us; skew 500 ppm
Sep 25 22:44:41 dist-test-slave-5vdr krb5kdc[14570](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376281, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.198.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:44:41.685017 14583 init.cc:377] Logged in from keytab as kudu/127.10.198.62@KRBTEST.COM (short username kudu)
I20260925 22:44:41.686487 14583 webserver.cc:460] Webserver started at http://127.10.198.62:43591/ using document root <none> and password file <none>
I20260925 22:44:41.687345 14583 fs_manager.cc:359] Metadata directory not provided
I20260925 22:44:41.687479 14583 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:44:41.687779 14583 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:44:41.691138 14583 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790376259650642-11032-0/minicluster-data/master-0/data/instance:
uuid: "ec8bef729c1a4f8c9458451a795ad996"
format_stamp: "Formatted at 2026-09-25 22:44:41 on dist-test-slave-5vdr"
server_key: "d42e0715c1e6aa86cc3f152954324479"
server_key_iv: "8703a7cab51ec8eda80b9480f0fa6737"
server_key_version: "encryptionkey@0"
I20260925 22:44:41.691920 14583 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790376259650642-11032-0/minicluster-data/master-0/wal/instance:
uuid: "ec8bef729c1a4f8c9458451a795ad996"
format_stamp: "Formatted at 2026-09-25 22:44:41 on dist-test-slave-5vdr"
server_key: "d42e0715c1e6aa86cc3f152954324479"
server_key_iv: "8703a7cab51ec8eda80b9480f0fa6737"
server_key_version: "encryptionkey@0"
I20260925 22:44:41.697932 14583 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.003s sys 0.002s
I20260925 22:44:41.702405 14598 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:44:41.704337 14583 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260925 22:44:41.704526 14583 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790376259650642-11032-0/minicluster-data/master-0/data,/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790376259650642-11032-0/minicluster-data/master-0/wal
uuid: "ec8bef729c1a4f8c9458451a795ad996"
format_stamp: "Formatted at 2026-09-25 22:44:41 on dist-test-slave-5vdr"
server_key: "d42e0715c1e6aa86cc3f152954324479"
server_key_iv: "8703a7cab51ec8eda80b9480f0fa6737"
server_key_version: "encryptionkey@0"
I20260925 22:44:41.704725 14583 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790376259650642-11032-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790376259650642-11032-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790376259650642-11032-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:44:41.735219 14583 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:44:41.739004 14583 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:44:41.739269 14583 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:44:41.747241 14583 rpc_server.cc:307] RPC server started. Bound to: 127.10.198.62:41515
I20260925 22:44:41.747260 14650 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.198.62:41515 every 8 connection(s)
I20260925 22:44:41.748414 14583 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790376259650642-11032-0/minicluster-data/master-0/data/info.pb
I20260925 22:44:41.751573 14651 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:44:41.754812 11032 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrWXm8X/build/debug/bin/kudu as pid 14583
I20260925 22:44:41.754962 11032 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790376259650642-11032-0/minicluster-data/master-0/wal/instance
I20260925 22:44:41.755276 11032 external_mini_cluster.cc:1428] Setting key fe042d3febcc80ace6153f037e186e53
I20260925 22:44:41.760818 14651 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ec8bef729c1a4f8c9458451a795ad996: Bootstrap starting.
I20260925 22:44:41.764098 14651 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ec8bef729c1a4f8c9458451a795ad996: Neither blocks nor log segments found. Creating new log.
I20260925 22:44:41.765201 14651 log.cc:824] T 00000000000000000000000000000000 P ec8bef729c1a4f8c9458451a795ad996: Log is configured to *not* fsync() on all Append() calls
Sep 25 22:44:41 dist-test-slave-5vdr krb5kdc[14570](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376281, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.198.62@KRBTEST.COM
I20260925 22:44:41.767966 14651 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ec8bef729c1a4f8c9458451a795ad996: No bootstrap required, opened a new log
I20260925 22:44:41.771772 14651 raft_consensus.cc:348] T 00000000000000000000000000000000 P ec8bef729c1a4f8c9458451a795ad996 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec8bef729c1a4f8c9458451a795ad996" member_type: VOTER last_known_addr { host: "127.10.198.62" port: 41515 } }
I20260925 22:44:41.772229 14651 raft_consensus.cc:374] T 00000000000000000000000000000000 P ec8bef729c1a4f8c9458451a795ad996 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:44:41.772318 14651 raft_consensus.cc:729] T 00000000000000000000000000000000 P ec8bef729c1a4f8c9458451a795ad996 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ec8bef729c1a4f8c9458451a795ad996, State: Initialized, Role: FOLLOWER
I20260925 22:44:41.772863 14651 consensus_queue.cc:260] T 00000000000000000000000000000000 P ec8bef729c1a4f8c9458451a795ad996 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec8bef729c1a4f8c9458451a795ad996" member_type: VOTER last_known_addr { host: "127.10.198.62" port: 41515 } }
I20260925 22:44:41.773012 14651 raft_consensus.cc:388] T 00000000000000000000000000000000 P ec8bef729c1a4f8c9458451a795ad996 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 22:44:41.773104 14651 raft_consensus.cc:482] T 00000000000000000000000000000000 P ec8bef729c1a4f8c9458451a795ad996 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 22:44:41.773211 14651 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ec8bef729c1a4f8c9458451a795ad996 [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:44:41.773394 14654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:44:41.757109 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39528 (local address 127.10.198.62:41515)
0925 22:44:41.757665 (+ 556us) server_negotiation.cc:206] Beginning negotiation
0925 22:44:41.757687 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0925 22:44:41.757737 (+ 50us) server_negotiation.cc:407] Connection header received
0925 22:44:41.758561 (+ 824us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:44:41.758623 (+ 62us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:44:41.759047 (+ 424us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:44:41.759408 (+ 361us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:44:41.760504 (+ 1096us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:44:41.761793 (+ 1289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:44:41.763122 (+ 1329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:44:41.763755 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:44:41.764014 (+ 259us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:44:41.767159 (+ 3145us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:44:41.767166 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:44:41.767183 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:44:41.767216 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:44:41.769644 (+ 2428us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:44:41.770269 (+ 625us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:44:41.770275 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:44:41.770281 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:44:41.770420 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:44:41.770748 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:44:41.770753 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:44:41.770755 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:44:41.771232 (+ 477us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:44:41.771488 (+ 256us) server_negotiation.cc:1035] Waiting for connection context
0925 22:44:41.771827 (+ 339us) server_negotiation.cc:299] Negotiation successful
0925 22:44:41.772279 (+ 452us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":358,"thread_start_us":160,"threads_started":1}
I20260925 22:44:41.774349 14651 raft_consensus.cc:504] T 00000000000000000000000000000000 P ec8bef729c1a4f8c9458451a795ad996 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec8bef729c1a4f8c9458451a795ad996" member_type: VOTER last_known_addr { host: "127.10.198.62" port: 41515 } }
I20260925 22:44:41.774782 14651 leader_election.cc:302] T 00000000000000000000000000000000 P ec8bef729c1a4f8c9458451a795ad996 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ec8bef729c1a4f8c9458451a795ad996; no voters:
I20260925 22:44:41.775195 14651 leader_election.cc:288] T 00000000000000000000000000000000 P ec8bef729c1a4f8c9458451a795ad996 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 22:44:41.775341 14656 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ec8bef729c1a4f8c9458451a795ad996 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 22:44:41.775648 14656 raft_consensus.cc:686] T 00000000000000000000000000000000 P ec8bef729c1a4f8c9458451a795ad996 [term 1 LEADER]: Becoming Leader. State: Replica: ec8bef729c1a4f8c9458451a795ad996, State: Running, Role: LEADER
I20260925 22:44:41.776358 14656 consensus_queue.cc:237] T 00000000000000000000000000000000 P ec8bef729c1a4f8c9458451a795ad996 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec8bef729c1a4f8c9458451a795ad996" member_type: VOTER last_known_addr { host: "127.10.198.62" port: 41515 } }
I20260925 22:44:41.776702 14651 sys_catalog.cc:564] T 00000000000000000000000000000000 P ec8bef729c1a4f8c9458451a795ad996 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 22:44:41.778642 14658 sys_catalog.cc:455] T 00000000000000000000000000000000 P ec8bef729c1a4f8c9458451a795ad996 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ec8bef729c1a4f8c9458451a795ad996. Latest consensus state: current_term: 1 leader_uuid: "ec8bef729c1a4f8c9458451a795ad996" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec8bef729c1a4f8c9458451a795ad996" member_type: VOTER last_known_addr { host: "127.10.198.62" port: 41515 } } }
I20260925 22:44:41.778843 14658 sys_catalog.cc:458] T 00000000000000000000000000000000 P ec8bef729c1a4f8c9458451a795ad996 [sys.catalog]: This master's current role is: LEADER
I20260925 22:44:41.778661 14657 sys_catalog.cc:455] T 00000000000000000000000000000000 P ec8bef729c1a4f8c9458451a795ad996 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ec8bef729c1a4f8c9458451a795ad996" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec8bef729c1a4f8c9458451a795ad996" member_type: VOTER last_known_addr { host: "127.10.198.62" port: 41515 } } }
I20260925 22:44:41.779983 14657 sys_catalog.cc:458] T 00000000000000000000000000000000 P ec8bef729c1a4f8c9458451a795ad996 [sys.catalog]: This master's current role is: LEADER
W20260925 22:44:41.781929 14670 catalog_manager.cc:1558] T 00000000000000000000000000000000 P ec8bef729c1a4f8c9458451a795ad996: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 22:44:41.782022 14670 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 22:44:41.782126 14671 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 22:44:41.782938 14671 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 22:44:41.790256 14671 catalog_manager.cc:1347] Generated new cluster ID: 5b427589d0eb40919be4b9fc7788e894
I20260925 22:44:41.790359 14671 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 22:44:41.812605 14671 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 22:44:41.814039 14671 catalog_manager.cc:1504] Loading token signing keys...
I20260925 22:44:41.824317 14671 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ec8bef729c1a4f8c9458451a795ad996: Generated new TSK 0
I20260925 22:44:41.825187 14671 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.10.198.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.198.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.198.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790376259650642-11032-0/krb5kdc/kudu_127.10.198.1.keytab.
Entry for principal kudu/127.10.198.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790376259650642-11032-0/krb5kdc/kudu_127.10.198.1.keytab.
I20260925 22:44:41.867714 11032 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrWXm8X/build/debug/bin/kudu
/tmp/dist-test-taskrWXm8X/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790376259650642-11032-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790376259650642-11032-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790376259650642-11032-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790376259650642-11032-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.198.1:0
--local_ip_for_outbound_sockets=127.10.198.1
--webserver_interface=127.10.198.1
--webserver_port=0
--tserver_master_addrs=127.10.198.62:41515
--keytab_file=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790376259650642-11032-0/krb5kdc/kudu_127.10.198.1.keytab
--principal=kudu/127.10.198.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.198.20:38063
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790376259650642-11032-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790376259650642-11032-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790376259650642-11032-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:44:42.013518 14677 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:44:42.013849 14677 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:44:42.013896 14677 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:44:42.018332 14677 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:44:42.018417 14677 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:44:42.018481 14677 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.198.1
I20260925 22:44:42.024188 14677 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.198.20:38063
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790376259650642-11032-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790376259650642-11032-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790376259650642-11032-0/krb5kdc/kudu_127.10.198.1.keytab
--principal=kudu/127.10.198.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.198.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790376259650642-11032-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.198.1
--webserver_port=0
--tserver_master_addrs=127.10.198.62:41515
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14677
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.198.1
--log_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790376259650642-11032-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:09:16 UTC on ccycloud.quasar-afmhyw.root.comops.site
build id 4673379
I20260925 22:44:42.025419 14677 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:44:42.026913 14677 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:44:42.037772 14685 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:44:42.037860 14683 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:44:42.037860 14682 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:44:42.038388 14677 server_base.cc:1034] running on GCE node
I20260925 22:44:42.038877 14677 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:44:42.039616 14677 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:44:42.040853 14677 hybrid_clock.cc:648] HybridClock initialized: now 1790376282040835 us; error 72 us; skew 500 ppm
Sep 25 22:44:42 dist-test-slave-5vdr krb5kdc[14570](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376282, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.198.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:44:42.044072 14677 init.cc:377] Logged in from keytab as kudu/127.10.198.1@KRBTEST.COM (short username kudu)
I20260925 22:44:42.045214 14677 webserver.cc:460] Webserver started at http://127.10.198.1:38339/ using document root <none> and password file <none>
I20260925 22:44:42.045887 14677 fs_manager.cc:359] Metadata directory not provided
I20260925 22:44:42.046000 14677 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:44:42.046262 14677 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:44:42.048771 14677 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790376259650642-11032-0/minicluster-data/ts-0/data/instance:
uuid: "1126db7b2d2f495890f0ab78fede3d7b"
format_stamp: "Formatted at 2026-09-25 22:44:42 on dist-test-slave-5vdr"
server_key: "96ae6e3fbba04b1a8602a2cb23bd96fd"
server_key_iv: "12843ad7e5c92deb8bcc5c690719e9b0"
server_key_version: "encryptionkey@0"
I20260925 22:44:42.049444 14677 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790376259650642-11032-0/minicluster-data/ts-0/wal/instance:
uuid: "1126db7b2d2f495890f0ab78fede3d7b"
format_stamp: "Formatted at 2026-09-25 22:44:42 on dist-test-slave-5vdr"
server_key: "96ae6e3fbba04b1a8602a2cb23bd96fd"
server_key_iv: "12843ad7e5c92deb8bcc5c690719e9b0"
server_key_version: "encryptionkey@0"
I20260925 22:44:42.053999 14677 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260925 22:44:42.056977 14692 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:44:42.058424 14677 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260925 22:44:42.058606 14677 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790376259650642-11032-0/minicluster-data/ts-0/data,/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790376259650642-11032-0/minicluster-data/ts-0/wal
uuid: "1126db7b2d2f495890f0ab78fede3d7b"
format_stamp: "Formatted at 2026-09-25 22:44:42 on dist-test-slave-5vdr"
server_key: "96ae6e3fbba04b1a8602a2cb23bd96fd"
server_key_iv: "12843ad7e5c92deb8bcc5c690719e9b0"
server_key_version: "encryptionkey@0"
I20260925 22:44:42.058756 14677 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790376259650642-11032-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790376259650642-11032-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790376259650642-11032-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:44:42.098119 14677 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:44:42.102006 14677 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:44:42.102260 14677 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:44:42.102977 14677 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 22:44:42.103938 14677 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:44:42.104014 14677 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:44:42.104084 14677 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:44:42.104135 14677 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 22:44:42.115685 14677 rpc_server.cc:307] RPC server started. Bound to: 127.10.198.1:36289
I20260925 22:44:42.115702 14805 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.198.1:36289 every 8 connection(s)
I20260925 22:44:42.116715 14677 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790376259650642-11032-0/minicluster-data/ts-0/data/info.pb
I20260925 22:44:42.117118 11032 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrWXm8X/build/debug/bin/kudu as pid 14677
I20260925 22:44:42.117271 11032 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790376259650642-11032-0/minicluster-data/ts-0/wal/instance
I20260925 22:44:42.117683 11032 external_mini_cluster.cc:1428] Setting key bc844415918a6130ac2888e10997bcd7
WARNING: no policy specified for kudu/127.10.198.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.198.2@KRBTEST.COM" created.
Sep 25 22:44:42 dist-test-slave-5vdr krb5kdc[14570](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376282, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.198.1@KRBTEST.COM for kudu/127.10.198.62@KRBTEST.COM
I20260925 22:44:42.138195 14654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:44:42.118867 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.198.1:56139 (local address 127.10.198.62:41515)
0925 22:44:42.119027 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0925 22:44:42.119032 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:44:42.120381 (+ 1349us) server_negotiation.cc:407] Connection header received
0925 22:44:42.122544 (+ 2163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:44:42.122550 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:44:42.122628 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:44:42.122735 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:44:42.124213 (+ 1478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:44:42.125138 (+ 925us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:44:42.126519 (+ 1381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:44:42.126941 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:44:42.127080 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:44:42.132710 (+ 5630us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:44:42.132721 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:44:42.132724 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:44:42.132757 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:44:42.135058 (+ 2301us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:44:42.135974 (+ 916us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:44:42.135978 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:44:42.135980 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:44:42.136087 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:44:42.136670 (+ 583us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:44:42.136674 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:44:42.136675 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:44:42.136948 (+ 273us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:44:42.137083 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0925 22:44:42.137766 (+ 683us) server_negotiation.cc:299] Negotiation successful
0925 22:44:42.137984 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260925 22:44:42.139536 14808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:44:42.119503 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.198.62:41515 (local address 127.10.198.1:56139)
0925 22:44:42.120173 (+ 670us) negotiation.cc:107] Waiting for socket to connect
0925 22:44:42.120243 (+ 70us) client_negotiation.cc:174] Beginning negotiation
0925 22:44:42.122213 (+ 1970us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:44:42.122931 (+ 718us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:44:42.122940 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:44:42.123372 (+ 432us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:44:42.124002 (+ 630us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:44:42.124020 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:44:42.125340 (+ 1320us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:44:42.125344 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:44:42.126345 (+ 1001us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:44:42.126357 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:44:42.127286 (+ 929us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:44:42.127291 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:44:42.127562 (+ 271us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:44:42.128565 (+ 1003us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:44:42.128587 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:44:42.132430 (+ 3843us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:44:42.135296 (+ 2866us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:44:42.135305 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:44:42.135336 (+ 31us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:44:42.135760 (+ 424us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:44:42.136190 (+ 430us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:44:42.136195 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:44:42.136197 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:44:42.136525 (+ 328us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:44:42.137078 (+ 553us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:44:42.137088 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:44:42.137467 (+ 379us) client_negotiation.cc:769] Sending connection context
0925 22:44:42.137715 (+ 248us) client_negotiation.cc:240] Negotiation successful
0925 22:44:42.138197 (+ 482us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":397,"thread_start_us":181,"threads_started":1}
I20260925 22:44:42.141420 14806 heartbeater.cc:347] Connected to a master server at 127.10.198.62:41515
I20260925 22:44:42.141736 14806 heartbeater.cc:464] Registering TS with master...
I20260925 22:44:42.142417 14806 heartbeater.cc:511] Master 127.10.198.62:41515 requested a full tablet report, sending...
I20260925 22:44:42.144476 14615 ts_manager.cc:194] Registered new tserver with Master: 1126db7b2d2f495890f0ab78fede3d7b (127.10.198.1:36289)
I20260925 22:44:42.145965 14615 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.198.1@KRBTEST.COM'} at 127.10.198.1:56139
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.198.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790376259650642-11032-0/krb5kdc/kudu_127.10.198.2.keytab.
Entry for principal kudu/127.10.198.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790376259650642-11032-0/krb5kdc/kudu_127.10.198.2.keytab.
I20260925 22:44:42.151464 11032 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrWXm8X/build/debug/bin/kudu
/tmp/dist-test-taskrWXm8X/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790376259650642-11032-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790376259650642-11032-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790376259650642-11032-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790376259650642-11032-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.198.2:0
--local_ip_for_outbound_sockets=127.10.198.2
--webserver_interface=127.10.198.2
--webserver_port=0
--tserver_master_addrs=127.10.198.62:41515
--keytab_file=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790376259650642-11032-0/krb5kdc/kudu_127.10.198.2.keytab
--principal=kudu/127.10.198.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.198.20:38063
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790376259650642-11032-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790376259650642-11032-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790376259650642-11032-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:44:42.272451 14811 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:44:42.272779 14811 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:44:42.272826 14811 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:44:42.277590 14811 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:44:42.277714 14811 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:44:42.277827 14811 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.198.2
I20260925 22:44:42.284119 14811 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.198.20:38063
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790376259650642-11032-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790376259650642-11032-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790376259650642-11032-0/krb5kdc/kudu_127.10.198.2.keytab
--principal=kudu/127.10.198.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.198.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790376259650642-11032-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.198.2
--webserver_port=0
--tserver_master_addrs=127.10.198.62:41515
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14811
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.198.2
--log_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790376259650642-11032-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:09:16 UTC on ccycloud.quasar-afmhyw.root.comops.site
build id 4673379
I20260925 22:44:42.285333 14811 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:44:42.286486 14811 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:44:42.297346 14817 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:44:42.297348 14819 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:44:42.297353 14816 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:44:42.297730 14811 server_base.cc:1034] running on GCE node
I20260925 22:44:42.298326 14811 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:44:42.299089 14811 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:44:42.300292 14811 hybrid_clock.cc:648] HybridClock initialized: now 1790376282300274 us; error 39 us; skew 500 ppm
Sep 25 22:44:42 dist-test-slave-5vdr krb5kdc[14570](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376282, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.198.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:44:42.303447 14811 init.cc:377] Logged in from keytab as kudu/127.10.198.2@KRBTEST.COM (short username kudu)
I20260925 22:44:42.304608 14811 webserver.cc:460] Webserver started at http://127.10.198.2:40299/ using document root <none> and password file <none>
I20260925 22:44:42.305265 14811 fs_manager.cc:359] Metadata directory not provided
I20260925 22:44:42.305372 14811 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:44:42.305629 14811 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:44:42.308074 14811 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790376259650642-11032-0/minicluster-data/ts-1/data/instance:
uuid: "d8c8df8887dd47c394f4cd234820aa25"
format_stamp: "Formatted at 2026-09-25 22:44:42 on dist-test-slave-5vdr"
server_key: "3bfb09518c5c4da80765a22fdf2d28de"
server_key_iv: "2bd286042dc7f8c8e272ed1f0cb8ee7c"
server_key_version: "encryptionkey@0"
I20260925 22:44:42.308841 14811 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790376259650642-11032-0/minicluster-data/ts-1/wal/instance:
uuid: "d8c8df8887dd47c394f4cd234820aa25"
format_stamp: "Formatted at 2026-09-25 22:44:42 on dist-test-slave-5vdr"
server_key: "3bfb09518c5c4da80765a22fdf2d28de"
server_key_iv: "2bd286042dc7f8c8e272ed1f0cb8ee7c"
server_key_version: "encryptionkey@0"
I20260925 22:44:42.313457 14811 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260925 22:44:42.316476 14826 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:44:42.317966 14811 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 22:44:42.318168 14811 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790376259650642-11032-0/minicluster-data/ts-1/data,/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790376259650642-11032-0/minicluster-data/ts-1/wal
uuid: "d8c8df8887dd47c394f4cd234820aa25"
format_stamp: "Formatted at 2026-09-25 22:44:42 on dist-test-slave-5vdr"
server_key: "3bfb09518c5c4da80765a22fdf2d28de"
server_key_iv: "2bd286042dc7f8c8e272ed1f0cb8ee7c"
server_key_version: "encryptionkey@0"
I20260925 22:44:42.318328 14811 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790376259650642-11032-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790376259650642-11032-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790376259650642-11032-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:44:42.333539 14811 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:44:42.337527 14811 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:44:42.337772 14811 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:44:42.338423 14811 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 22:44:42.339358 14811 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:44:42.339443 14811 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:44:42.339547 14811 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:44:42.339598 14811 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 22:44:42.349489 14811 rpc_server.cc:307] RPC server started. Bound to: 127.10.198.2:42271
I20260925 22:44:42.349555 14939 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.198.2:42271 every 8 connection(s)
I20260925 22:44:42.350579 14811 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790376259650642-11032-0/minicluster-data/ts-1/data/info.pb
I20260925 22:44:42.359556 11032 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrWXm8X/build/debug/bin/kudu as pid 14811
I20260925 22:44:42.359719 11032 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790376259650642-11032-0/minicluster-data/ts-1/wal/instance
I20260925 22:44:42.360080 11032 external_mini_cluster.cc:1428] Setting key 11d1237ba67667822d4f8805f50702f4
Sep 25 22:44:42 dist-test-slave-5vdr krb5kdc[14570](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376282, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.198.2@KRBTEST.COM for kudu/127.10.198.62@KRBTEST.COM
I20260925 22:44:42.367323 14654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:44:42.352907 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.198.2:60799 (local address 127.10.198.62:41515)
0925 22:44:42.353077 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0925 22:44:42.353081 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:44:42.354068 (+ 987us) server_negotiation.cc:407] Connection header received
0925 22:44:42.355570 (+ 1502us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:44:42.355574 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:44:42.355637 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:44:42.355734 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:44:42.357081 (+ 1347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:44:42.357630 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:44:42.358909 (+ 1279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:44:42.359146 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:44:42.359271 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:44:42.362543 (+ 3272us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:44:42.362547 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:44:42.362550 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:44:42.362584 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:44:42.364570 (+ 1986us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:44:42.365305 (+ 735us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:44:42.365309 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:44:42.365311 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:44:42.365409 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:44:42.365892 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:44:42.365894 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:44:42.365896 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:44:42.366118 (+ 222us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:44:42.366204 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0925 22:44:42.367028 (+ 824us) server_negotiation.cc:299] Negotiation successful
0925 22:44:42.367178 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260925 22:44:42.368789 14942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:44:42.353362 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.198.62:41515 (local address 127.10.198.2:60799)
0925 22:44:42.353887 (+ 525us) negotiation.cc:107] Waiting for socket to connect
0925 22:44:42.353936 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0925 22:44:42.355312 (+ 1376us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:44:42.355968 (+ 656us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:44:42.355977 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:44:42.356387 (+ 410us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:44:42.356910 (+ 523us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:44:42.356925 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:44:42.357798 (+ 873us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:44:42.357802 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:44:42.358759 (+ 957us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:44:42.358769 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:44:42.359325 (+ 556us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:44:42.359330 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:44:42.359515 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:44:42.360291 (+ 776us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:44:42.360310 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:44:42.362327 (+ 2017us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:44:42.364769 (+ 2442us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:44:42.364776 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:44:42.364796 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:44:42.365154 (+ 358us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:44:42.365519 (+ 365us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:44:42.365525 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:44:42.365530 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:44:42.365781 (+ 251us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:44:42.366237 (+ 456us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:44:42.366248 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:44:42.366676 (+ 428us) client_negotiation.cc:769] Sending connection context
0925 22:44:42.366949 (+ 273us) client_negotiation.cc:240] Negotiation successful
0925 22:44:42.367394 (+ 445us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":318,"thread_start_us":134,"threads_started":1}
I20260925 22:44:42.370474 14940 heartbeater.cc:347] Connected to a master server at 127.10.198.62:41515
I20260925 22:44:42.370785 14940 heartbeater.cc:464] Registering TS with master...
I20260925 22:44:42.371373 14940 heartbeater.cc:511] Master 127.10.198.62:41515 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.10.198.3@KRBTEST.COM; defaulting to no policy
I20260925 22:44:42.372774 14615 ts_manager.cc:194] Registered new tserver with Master: d8c8df8887dd47c394f4cd234820aa25 (127.10.198.2:42271)
I20260925 22:44:42.373405 14615 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.198.2@KRBTEST.COM'} at 127.10.198.2:60799
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.198.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.198.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790376259650642-11032-0/krb5kdc/kudu_127.10.198.3.keytab.
Entry for principal kudu/127.10.198.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790376259650642-11032-0/krb5kdc/kudu_127.10.198.3.keytab.
I20260925 22:44:42.390941 11032 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrWXm8X/build/debug/bin/kudu
/tmp/dist-test-taskrWXm8X/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790376259650642-11032-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790376259650642-11032-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790376259650642-11032-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790376259650642-11032-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.198.3:0
--local_ip_for_outbound_sockets=127.10.198.3
--webserver_interface=127.10.198.3
--webserver_port=0
--tserver_master_addrs=127.10.198.62:41515
--keytab_file=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790376259650642-11032-0/krb5kdc/kudu_127.10.198.3.keytab
--principal=kudu/127.10.198.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.198.20:38063
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790376259650642-11032-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790376259650642-11032-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790376259650642-11032-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:44:42.514276 14945 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:44:42.514652 14945 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:44:42.514732 14945 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:44:42.519565 14945 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:44:42.519713 14945 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:44:42.519821 14945 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.198.3
I20260925 22:44:42.526165 14945 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.198.20:38063
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790376259650642-11032-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790376259650642-11032-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790376259650642-11032-0/krb5kdc/kudu_127.10.198.3.keytab
--principal=kudu/127.10.198.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.198.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790376259650642-11032-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.198.3
--webserver_port=0
--tserver_master_addrs=127.10.198.62:41515
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14945
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.198.3
--log_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790376259650642-11032-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:09:16 UTC on ccycloud.quasar-afmhyw.root.comops.site
build id 4673379
I20260925 22:44:42.527390 14945 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:44:42.528596 14945 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:44:42.539031 14950 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:44:42.539031 14951 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:44:42.539033 14953 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:44:42.539474 14945 server_base.cc:1034] running on GCE node
I20260925 22:44:42.540035 14945 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:44:42.540812 14945 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:44:42.542057 14945 hybrid_clock.cc:648] HybridClock initialized: now 1790376282542023 us; error 57 us; skew 500 ppm
Sep 25 22:44:42 dist-test-slave-5vdr krb5kdc[14570](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376282, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.198.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:44:42.545231 14945 init.cc:377] Logged in from keytab as kudu/127.10.198.3@KRBTEST.COM (short username kudu)
I20260925 22:44:42.546289 14945 webserver.cc:460] Webserver started at http://127.10.198.3:36191/ using document root <none> and password file <none>
I20260925 22:44:42.546902 14945 fs_manager.cc:359] Metadata directory not provided
I20260925 22:44:42.546976 14945 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:44:42.547214 14945 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:44:42.549605 14945 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790376259650642-11032-0/minicluster-data/ts-2/data/instance:
uuid: "3e029c1a9a66492abfa08e115cf81cc1"
format_stamp: "Formatted at 2026-09-25 22:44:42 on dist-test-slave-5vdr"
server_key: "e36061501cedae396687b54517a07a2e"
server_key_iv: "477d50fd8ff60152052d204bcdcb2bda"
server_key_version: "encryptionkey@0"
I20260925 22:44:42.550283 14945 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790376259650642-11032-0/minicluster-data/ts-2/wal/instance:
uuid: "3e029c1a9a66492abfa08e115cf81cc1"
format_stamp: "Formatted at 2026-09-25 22:44:42 on dist-test-slave-5vdr"
server_key: "e36061501cedae396687b54517a07a2e"
server_key_iv: "477d50fd8ff60152052d204bcdcb2bda"
server_key_version: "encryptionkey@0"
I20260925 22:44:42.554507 14945 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260925 22:44:42.557674 14960 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:44:42.559288 14945 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260925 22:44:42.559466 14945 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790376259650642-11032-0/minicluster-data/ts-2/data,/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790376259650642-11032-0/minicluster-data/ts-2/wal
uuid: "3e029c1a9a66492abfa08e115cf81cc1"
format_stamp: "Formatted at 2026-09-25 22:44:42 on dist-test-slave-5vdr"
server_key: "e36061501cedae396687b54517a07a2e"
server_key_iv: "477d50fd8ff60152052d204bcdcb2bda"
server_key_version: "encryptionkey@0"
I20260925 22:44:42.559614 14945 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790376259650642-11032-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790376259650642-11032-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790376259650642-11032-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:44:42.579320 14945 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:44:42.582998 14945 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:44:42.583256 14945 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:44:42.584018 14945 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 22:44:42.585156 14945 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:44:42.585243 14945 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:44:42.585323 14945 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:44:42.585369 14945 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 22:44:42.596412 14945 rpc_server.cc:307] RPC server started. Bound to: 127.10.198.3:35681
I20260925 22:44:42.596439 15073 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.198.3:35681 every 8 connection(s)
I20260925 22:44:42.597625 14945 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790376259650642-11032-0/minicluster-data/ts-2/data/info.pb
I20260925 22:44:42.599946 11032 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrWXm8X/build/debug/bin/kudu as pid 14945
I20260925 22:44:42.600067 11032 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790376259650642-11032-0/minicluster-data/ts-2/wal/instance
I20260925 22:44:42.600427 11032 external_mini_cluster.cc:1428] Setting key c94a4b7a36c784134cad9f6f3d8a5004
Sep 25 22:44:42 dist-test-slave-5vdr krb5kdc[14570](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376282, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.198.3@KRBTEST.COM for kudu/127.10.198.62@KRBTEST.COM
I20260925 22:44:42.615083 14654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:44:42.600287 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.198.3:44267 (local address 127.10.198.62:41515)
0925 22:44:42.600467 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0925 22:44:42.600472 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:44:42.601608 (+ 1136us) server_negotiation.cc:407] Connection header received
0925 22:44:42.603304 (+ 1696us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:44:42.603306 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:44:42.603354 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:44:42.603423 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:44:42.604812 (+ 1389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:44:42.605358 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:44:42.606643 (+ 1285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:44:42.606886 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:44:42.606980 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:44:42.610640 (+ 3660us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:44:42.610644 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:44:42.610646 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:44:42.610685 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:44:42.612220 (+ 1535us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:44:42.612894 (+ 674us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:44:42.612900 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:44:42.612902 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:44:42.613014 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:44:42.613609 (+ 595us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:44:42.613612 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:44:42.613614 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:44:42.613912 (+ 298us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:44:42.614019 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0925 22:44:42.614732 (+ 713us) server_negotiation.cc:299] Negotiation successful
0925 22:44:42.614921 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260925 22:44:42.616184 15076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:44:42.600820 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.198.62:41515 (local address 127.10.198.3:44267)
0925 22:44:42.601437 (+ 617us) negotiation.cc:107] Waiting for socket to connect
0925 22:44:42.601507 (+ 70us) client_negotiation.cc:174] Beginning negotiation
0925 22:44:42.603074 (+ 1567us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:44:42.603576 (+ 502us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:44:42.603583 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:44:42.604091 (+ 508us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:44:42.604634 (+ 543us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:44:42.604651 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:44:42.605496 (+ 845us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:44:42.605499 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:44:42.606481 (+ 982us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:44:42.606493 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:44:42.607044 (+ 551us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:44:42.607047 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:44:42.607225 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:44:42.607966 (+ 741us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:44:42.607984 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:44:42.610353 (+ 2369us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:44:42.612386 (+ 2033us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:44:42.612400 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:44:42.612423 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:44:42.612759 (+ 336us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:44:42.613163 (+ 404us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:44:42.613166 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:44:42.613168 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:44:42.613477 (+ 309us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:44:42.614043 (+ 566us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:44:42.614053 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:44:42.614409 (+ 356us) client_negotiation.cc:769] Sending connection context
0925 22:44:42.614676 (+ 267us) client_negotiation.cc:240] Negotiation successful
0925 22:44:42.615115 (+ 439us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":354,"thread_start_us":150,"threads_started":1}
I20260925 22:44:42.617615 15074 heartbeater.cc:347] Connected to a master server at 127.10.198.62:41515
I20260925 22:44:42.617877 15074 heartbeater.cc:464] Registering TS with master...
I20260925 22:44:42.618511 15074 heartbeater.cc:511] Master 127.10.198.62:41515 requested a full tablet report, sending...
I20260925 22:44:42.619737 14615 ts_manager.cc:194] Registered new tserver with Master: 3e029c1a9a66492abfa08e115cf81cc1 (127.10.198.3:35681)
I20260925 22:44:42.620440 14615 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.198.3@KRBTEST.COM'} at 127.10.198.3:44267
I20260925 22:44:42.627564 11032 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 25 22:44:42 dist-test-slave-5vdr krb5kdc[14570](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376282, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 25 22:44:42 dist-test-slave-5vdr krb5kdc[14570](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376282, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.198.1@KRBTEST.COM
I20260925 22:44:42.656440 15081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:44:42.645075 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43854 (local address 127.10.198.1:36289)
0925 22:44:42.645398 (+ 323us) server_negotiation.cc:206] Beginning negotiation
0925 22:44:42.645404 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 22:44:42.645430 (+ 26us) server_negotiation.cc:407] Connection header received
0925 22:44:42.645543 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:44:42.645548 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:44:42.645743 (+ 195us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:44:42.645874 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:44:42.646694 (+ 820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:44:42.647501 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:44:42.648512 (+ 1011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:44:42.648923 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:44:42.649017 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:44:42.651745 (+ 2728us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:44:42.651758 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:44:42.651776 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:44:42.651808 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:44:42.653913 (+ 2105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:44:42.654542 (+ 629us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:44:42.654549 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:44:42.654550 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:44:42.654663 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:44:42.654985 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:44:42.654987 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:44:42.654989 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:44:42.655308 (+ 319us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:44:42.655512 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0925 22:44:42.656004 (+ 492us) server_negotiation.cc:299] Negotiation successful
0925 22:44:42.656232 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":198,"thread_start_us":102,"threads_started":1}
Sep 25 22:44:42 dist-test-slave-5vdr krb5kdc[14570](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376282, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.198.62@KRBTEST.COM
I20260925 22:44:42.669283 14654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:44:42.659795 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39530 (local address 127.10.198.62:41515)
0925 22:44:42.659973 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0925 22:44:42.659977 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:44:42.660122 (+ 145us) server_negotiation.cc:407] Connection header received
0925 22:44:42.660234 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:44:42.660237 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:44:42.660281 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:44:42.660352 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:44:42.661023 (+ 671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:44:42.661552 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:44:42.662501 (+ 949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:44:42.662809 (+ 308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:44:42.662894 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:44:42.665674 (+ 2780us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:44:42.665677 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:44:42.665679 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:44:42.665714 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:44:42.667289 (+ 1575us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:44:42.667784 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:44:42.667787 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:44:42.667788 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:44:42.668001 (+ 213us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:44:42.668394 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:44:42.668397 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:44:42.668398 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:44:42.668658 (+ 260us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:44:42.668776 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0925 22:44:42.669010 (+ 234us) server_negotiation.cc:299] Negotiation successful
0925 22:44:42.669149 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91}
W20260925 22:44:42.669775 14615 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39530
I20260925 22:44:42.670531 11032 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrWXm8X/build/debug/bin/kudu with pid 14677
I20260925 22:44:42.676708 11032 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrWXm8X/build/debug/bin/kudu with pid 14811
I20260925 22:44:42.682498 11032 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrWXm8X/build/debug/bin/kudu with pid 14945
I20260925 22:44:42.687933 11032 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrWXm8X/build/debug/bin/kudu with pid 14583
2026-09-25T22:44:42Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (2347 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790376259650642-11032-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 22:44:42 dist-test-slave-5vdr krb5kdc[15086](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 22:44:42 dist-test-slave-5vdr krb5kdc[15086](info): set up 2 sockets
Sep 25 22:44:42 dist-test-slave-5vdr krb5kdc[15086](info): commencing operation
krb5kdc: starting...
W20260925 22:44:43.780119 11032 mini_kdc.cc:121] Time spent starting KDC: real 1.069s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790376259650642-11032-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790376259650642-11032-0/krb5kdc/test-user.keytab.
Sep 25 22:44:43 dist-test-slave-5vdr krb5kdc[15086](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376283, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T22:44:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T22:44:43Z Disabled control of system clock
WARNING: no policy specified for kudu/127.10.198.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.198.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.198.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790376259650642-11032-0/krb5kdc/kudu_127.10.198.62.keytab.
Entry for principal kudu/127.10.198.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790376259650642-11032-0/krb5kdc/kudu_127.10.198.62.keytab.
I20260925 22:44:43.912339 11032 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrWXm8X/build/debug/bin/kudu
/tmp/dist-test-taskrWXm8X/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790376259650642-11032-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790376259650642-11032-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790376259650642-11032-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790376259650642-11032-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.198.62:40365
--webserver_interface=127.10.198.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790376259650642-11032-0/krb5kdc/kudu_127.10.198.62.keytab
--principal=kudu/127.10.198.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.198.20:46367
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.198.62:40365
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790376259650642-11032-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790376259650642-11032-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790376259650642-11032-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:44:44.032402 15099 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:44:44.032735 15099 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:44:44.032783 15099 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:44:44.037173 15099 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 22:44:44.037262 15099 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:44:44.037283 15099 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:44:44.037302 15099 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 22:44:44.037317 15099 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 22:44:44.043409 15099 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.198.20:46367
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790376259650642-11032-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790376259650642-11032-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.198.62:40365
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790376259650642-11032-0/krb5kdc/kudu_127.10.198.62.keytab
--principal=kudu/127.10.198.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.198.62:40365
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790376259650642-11032-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.198.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15099
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790376259650642-11032-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:09:16 UTC on ccycloud.quasar-afmhyw.root.comops.site
build id 4673379
I20260925 22:44:44.044704 15099 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:44:44.045852 15099 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:44:44.055171 15107 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:44:44.055172 15105 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:44:44.055171 15104 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:44:44.055233 15099 server_base.cc:1034] running on GCE node
I20260925 22:44:44.056257 15099 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:44:44.057602 15099 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:44:44.058835 15099 hybrid_clock.cc:648] HybridClock initialized: now 1790376284058815 us; error 52 us; skew 500 ppm
Sep 25 22:44:44 dist-test-slave-5vdr krb5kdc[15086](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376284, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.198.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:44:44.062366 15099 init.cc:377] Logged in from keytab as kudu/127.10.198.62@KRBTEST.COM (short username kudu)
I20260925 22:44:44.063259 15099 fs_manager.cc:359] Metadata directory not provided
I20260925 22:44:44.063371 15099 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:44:44.063639 15099 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:44:44.066152 15099 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790376259650642-11032-0/minicluster-data/master-0/data/instance:
uuid: "87b1e94f8c0f4aa0b240b08515856297"
format_stamp: "Formatted at 2026-09-25 22:44:44 on dist-test-slave-5vdr"
server_key: "664748d77913ab43800f8b1e550176b0"
server_key_iv: "83f4f1266d1c02c5a82da937723d7bcb"
server_key_version: "encryptionkey@0"
I20260925 22:44:44.066862 15099 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790376259650642-11032-0/minicluster-data/master-0/wal/instance:
uuid: "87b1e94f8c0f4aa0b240b08515856297"
format_stamp: "Formatted at 2026-09-25 22:44:44 on dist-test-slave-5vdr"
server_key: "664748d77913ab43800f8b1e550176b0"
server_key_iv: "83f4f1266d1c02c5a82da937723d7bcb"
server_key_version: "encryptionkey@0"
I20260925 22:44:44.071774 15099 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260925 22:44:44.074791 15113 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:44:44.076490 15099 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 22:44:44.076712 15099 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790376259650642-11032-0/minicluster-data/master-0/data,/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790376259650642-11032-0/minicluster-data/master-0/wal
uuid: "87b1e94f8c0f4aa0b240b08515856297"
format_stamp: "Formatted at 2026-09-25 22:44:44 on dist-test-slave-5vdr"
server_key: "664748d77913ab43800f8b1e550176b0"
server_key_iv: "83f4f1266d1c02c5a82da937723d7bcb"
server_key_version: "encryptionkey@0"
I20260925 22:44:44.076876 15099 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790376259650642-11032-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790376259650642-11032-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790376259650642-11032-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:44:44.104796 15099 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:44:44.108644 15099 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:44:44.108894 15099 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:44:44.116801 15099 rpc_server.cc:307] RPC server started. Bound to: 127.10.198.62:40365
I20260925 22:44:44.117136 15165 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.198.62:40365 every 8 connection(s)
I20260925 22:44:44.117632 15099 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790376259650642-11032-0/minicluster-data/master-0/data/info.pb
I20260925 22:44:44.120484 11032 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrWXm8X/build/debug/bin/kudu as pid 15099
I20260925 22:44:44.120667 11032 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790376259650642-11032-0/minicluster-data/master-0/wal/instance
I20260925 22:44:44.120987 11032 external_mini_cluster.cc:1428] Setting key 4c6d62fd53398169aa25a1347f2b5c9a
I20260925 22:44:44.120977 15166 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:44:44.128376 15166 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 87b1e94f8c0f4aa0b240b08515856297: Bootstrap starting.
Sep 25 22:44:44 dist-test-slave-5vdr krb5kdc[15086](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376283, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.198.62@KRBTEST.COM
I20260925 22:44:44.131412 15166 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 87b1e94f8c0f4aa0b240b08515856297: Neither blocks nor log segments found. Creating new log.
I20260925 22:44:44.132357 15166 log.cc:824] T 00000000000000000000000000000000 P 87b1e94f8c0f4aa0b240b08515856297: Log is configured to *not* fsync() on all Append() calls
I20260925 22:44:44.135288 15166 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 87b1e94f8c0f4aa0b240b08515856297: No bootstrap required, opened a new log
I20260925 22:44:44.138489 15169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:44:44.122510 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55306 (local address 127.10.198.62:40365)
0925 22:44:44.123044 (+ 534us) server_negotiation.cc:206] Beginning negotiation
0925 22:44:44.123061 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0925 22:44:44.123101 (+ 40us) server_negotiation.cc:407] Connection header received
0925 22:44:44.124174 (+ 1073us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:44:44.124200 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:44:44.124645 (+ 445us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:44:44.125037 (+ 392us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:44:44.125951 (+ 914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:44:44.127034 (+ 1083us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:44:44.128053 (+ 1019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:44:44.128711 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:44:44.128912 (+ 201us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:44:44.132025 (+ 3113us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:44:44.132032 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:44:44.132050 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:44:44.132090 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:44:44.134478 (+ 2388us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:44:44.135130 (+ 652us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:44:44.135136 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:44:44.135142 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:44:44.135310 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:44:44.135744 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:44:44.135748 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:44:44.135751 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:44:44.136407 (+ 656us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:44:44.136624 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0925 22:44:44.136917 (+ 293us) server_negotiation.cc:299] Negotiation successful
0925 22:44:44.137272 (+ 355us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":367,"thread_start_us":165,"threads_started":1}
I20260925 22:44:44.140729 15166 raft_consensus.cc:348] T 00000000000000000000000000000000 P 87b1e94f8c0f4aa0b240b08515856297 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "87b1e94f8c0f4aa0b240b08515856297" member_type: VOTER last_known_addr { host: "127.10.198.62" port: 40365 } }
I20260925 22:44:44.141109 15166 raft_consensus.cc:374] T 00000000000000000000000000000000 P 87b1e94f8c0f4aa0b240b08515856297 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:44:44.141165 15166 raft_consensus.cc:729] T 00000000000000000000000000000000 P 87b1e94f8c0f4aa0b240b08515856297 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 87b1e94f8c0f4aa0b240b08515856297, State: Initialized, Role: FOLLOWER
I20260925 22:44:44.141690 15166 consensus_queue.cc:260] T 00000000000000000000000000000000 P 87b1e94f8c0f4aa0b240b08515856297 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "87b1e94f8c0f4aa0b240b08515856297" member_type: VOTER last_known_addr { host: "127.10.198.62" port: 40365 } }
I20260925 22:44:44.141898 15166 raft_consensus.cc:388] T 00000000000000000000000000000000 P 87b1e94f8c0f4aa0b240b08515856297 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 22:44:44.142020 15166 raft_consensus.cc:482] T 00000000000000000000000000000000 P 87b1e94f8c0f4aa0b240b08515856297 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 22:44:44.142158 15166 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 87b1e94f8c0f4aa0b240b08515856297 [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:44:44.143483 15166 raft_consensus.cc:504] T 00000000000000000000000000000000 P 87b1e94f8c0f4aa0b240b08515856297 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "87b1e94f8c0f4aa0b240b08515856297" member_type: VOTER last_known_addr { host: "127.10.198.62" port: 40365 } }
I20260925 22:44:44.144003 15166 leader_election.cc:302] T 00000000000000000000000000000000 P 87b1e94f8c0f4aa0b240b08515856297 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 87b1e94f8c0f4aa0b240b08515856297; no voters:
I20260925 22:44:44.144433 15166 leader_election.cc:288] T 00000000000000000000000000000000 P 87b1e94f8c0f4aa0b240b08515856297 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 22:44:44.144572 15171 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 87b1e94f8c0f4aa0b240b08515856297 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 22:44:44.144826 15171 raft_consensus.cc:686] T 00000000000000000000000000000000 P 87b1e94f8c0f4aa0b240b08515856297 [term 1 LEADER]: Becoming Leader. State: Replica: 87b1e94f8c0f4aa0b240b08515856297, State: Running, Role: LEADER
I20260925 22:44:44.145269 15171 consensus_queue.cc:237] T 00000000000000000000000000000000 P 87b1e94f8c0f4aa0b240b08515856297 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "87b1e94f8c0f4aa0b240b08515856297" member_type: VOTER last_known_addr { host: "127.10.198.62" port: 40365 } }
I20260925 22:44:44.146139 15166 sys_catalog.cc:564] T 00000000000000000000000000000000 P 87b1e94f8c0f4aa0b240b08515856297 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 22:44:44.147158 15172 sys_catalog.cc:455] T 00000000000000000000000000000000 P 87b1e94f8c0f4aa0b240b08515856297 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "87b1e94f8c0f4aa0b240b08515856297" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "87b1e94f8c0f4aa0b240b08515856297" member_type: VOTER last_known_addr { host: "127.10.198.62" port: 40365 } } }
I20260925 22:44:44.147312 15172 sys_catalog.cc:458] T 00000000000000000000000000000000 P 87b1e94f8c0f4aa0b240b08515856297 [sys.catalog]: This master's current role is: LEADER
I20260925 22:44:44.147459 15173 sys_catalog.cc:455] T 00000000000000000000000000000000 P 87b1e94f8c0f4aa0b240b08515856297 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 87b1e94f8c0f4aa0b240b08515856297. Latest consensus state: current_term: 1 leader_uuid: "87b1e94f8c0f4aa0b240b08515856297" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "87b1e94f8c0f4aa0b240b08515856297" member_type: VOTER last_known_addr { host: "127.10.198.62" port: 40365 } } }
I20260925 22:44:44.147590 15173 sys_catalog.cc:458] T 00000000000000000000000000000000 P 87b1e94f8c0f4aa0b240b08515856297 [sys.catalog]: This master's current role is: LEADER
I20260925 22:44:44.148002 15180 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 22:44:44.151196 15180 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 22:44:44.157485 15180 catalog_manager.cc:1347] Generated new cluster ID: ad0cd602000145c18b55aeac19acb4b2
I20260925 22:44:44.157586 15180 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 22:44:44.164186 15180 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 22:44:44.165254 15180 catalog_manager.cc:1504] Loading token signing keys...
I20260925 22:44:44.175120 15180 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 87b1e94f8c0f4aa0b240b08515856297: Generated new TSK 0
I20260925 22:44:44.176074 15180 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.10.198.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.198.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.198.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790376259650642-11032-0/krb5kdc/kudu_127.10.198.1.keytab.
Entry for principal kudu/127.10.198.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790376259650642-11032-0/krb5kdc/kudu_127.10.198.1.keytab.
I20260925 22:44:44.207492 11032 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrWXm8X/build/debug/bin/kudu
/tmp/dist-test-taskrWXm8X/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790376259650642-11032-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790376259650642-11032-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790376259650642-11032-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790376259650642-11032-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.198.1:0
--local_ip_for_outbound_sockets=127.10.198.1
--webserver_interface=127.10.198.1
--webserver_port=0
--tserver_master_addrs=127.10.198.62:40365
--keytab_file=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790376259650642-11032-0/krb5kdc/kudu_127.10.198.1.keytab
--principal=kudu/127.10.198.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.198.20:46367
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790376259650642-11032-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790376259650642-11032-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790376259650642-11032-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:44:44.332998 15192 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:44:44.333320 15192 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:44:44.333369 15192 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:44:44.337898 15192 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:44:44.337988 15192 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:44:44.338056 15192 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.198.1
I20260925 22:44:44.344166 15192 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.198.20:46367
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790376259650642-11032-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790376259650642-11032-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790376259650642-11032-0/krb5kdc/kudu_127.10.198.1.keytab
--principal=kudu/127.10.198.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.198.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790376259650642-11032-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.198.1
--webserver_port=0
--tserver_master_addrs=127.10.198.62:40365
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15192
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.198.1
--log_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790376259650642-11032-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:09:16 UTC on ccycloud.quasar-afmhyw.root.comops.site
build id 4673379
I20260925 22:44:44.345472 15192 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:44:44.346632 15192 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:44:44.357368 15198 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:44:44.357555 15192 server_base.cc:1034] running on GCE node
W20260925 22:44:44.357368 15197 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:44:44.357816 15200 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:44:44.358320 15192 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:44:44.359073 15192 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:44:44.360256 15192 hybrid_clock.cc:648] HybridClock initialized: now 1790376284360244 us; error 35 us; skew 500 ppm
Sep 25 22:44:44 dist-test-slave-5vdr krb5kdc[15086](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376284, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.198.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:44:44.363525 15192 init.cc:377] Logged in from keytab as kudu/127.10.198.1@KRBTEST.COM (short username kudu)
I20260925 22:44:44.364488 15192 fs_manager.cc:359] Metadata directory not provided
I20260925 22:44:44.364579 15192 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:44:44.364861 15192 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:44:44.367246 15192 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790376259650642-11032-0/minicluster-data/ts-0/data/instance:
uuid: "117d64e10a9b4976a84373ccf4851c0e"
format_stamp: "Formatted at 2026-09-25 22:44:44 on dist-test-slave-5vdr"
server_key: "06c9d717ed65528f52f457216f6b2d2d"
server_key_iv: "a271980d78876246c0f46a07cf380f69"
server_key_version: "encryptionkey@0"
I20260925 22:44:44.368033 15192 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790376259650642-11032-0/minicluster-data/ts-0/wal/instance:
uuid: "117d64e10a9b4976a84373ccf4851c0e"
format_stamp: "Formatted at 2026-09-25 22:44:44 on dist-test-slave-5vdr"
server_key: "06c9d717ed65528f52f457216f6b2d2d"
server_key_iv: "a271980d78876246c0f46a07cf380f69"
server_key_version: "encryptionkey@0"
I20260925 22:44:44.372521 15192 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260925 22:44:44.376039 15206 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:44:44.377648 15192 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 22:44:44.377859 15192 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790376259650642-11032-0/minicluster-data/ts-0/data,/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790376259650642-11032-0/minicluster-data/ts-0/wal
uuid: "117d64e10a9b4976a84373ccf4851c0e"
format_stamp: "Formatted at 2026-09-25 22:44:44 on dist-test-slave-5vdr"
server_key: "06c9d717ed65528f52f457216f6b2d2d"
server_key_iv: "a271980d78876246c0f46a07cf380f69"
server_key_version: "encryptionkey@0"
I20260925 22:44:44.378026 15192 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790376259650642-11032-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790376259650642-11032-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790376259650642-11032-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:44:44.407383 15192 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:44:44.412016 15192 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:44:44.412269 15192 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:44:44.413167 15192 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 22:44:44.414232 15192 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:44:44.414320 15192 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:44:44.414397 15192 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:44:44.414451 15192 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 22:44:44.424691 15192 rpc_server.cc:307] RPC server started. Bound to: 127.10.198.1:43219
I20260925 22:44:44.424706 15319 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.198.1:43219 every 8 connection(s)
I20260925 22:44:44.425377 15192 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790376259650642-11032-0/minicluster-data/ts-0/data/info.pb
I20260925 22:44:44.426162 11032 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrWXm8X/build/debug/bin/kudu as pid 15192
I20260925 22:44:44.426411 11032 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790376259650642-11032-0/minicluster-data/ts-0/wal/instance
I20260925 22:44:44.426848 11032 external_mini_cluster.cc:1428] Setting key 2ce3fd3dc74f78a578de7d0b45410707
WARNING: no policy specified for kudu/127.10.198.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.198.2@KRBTEST.COM" created.
Sep 25 22:44:44 dist-test-slave-5vdr krb5kdc[15086](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376284, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.198.1@KRBTEST.COM for kudu/127.10.198.62@KRBTEST.COM
I20260925 22:44:44.446002 15169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:44:44.428033 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.198.1:51555 (local address 127.10.198.62:40365)
0925 22:44:44.428177 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0925 22:44:44.428180 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 22:44:44.429396 (+ 1216us) server_negotiation.cc:407] Connection header received
0925 22:44:44.431243 (+ 1847us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:44:44.431247 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:44:44.431328 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:44:44.431417 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:44:44.432920 (+ 1503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:44:44.433486 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:44:44.434830 (+ 1344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:44:44.435093 (+ 263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:44:44.435228 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:44:44.441010 (+ 5782us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:44:44.441018 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:44:44.441025 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:44:44.441070 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:44:44.442876 (+ 1806us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:44:44.443727 (+ 851us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:44:44.443730 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:44:44.443731 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:44:44.443823 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:44:44.444429 (+ 606us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:44:44.444432 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:44:44.444433 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:44:44.444661 (+ 228us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:44:44.444751 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0925 22:44:44.445584 (+ 833us) server_negotiation.cc:299] Negotiation successful
0925 22:44:44.445748 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260925 22:44:44.447443 15322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:44:44.428635 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.198.62:40365 (local address 127.10.198.1:51555)
0925 22:44:44.429207 (+ 572us) negotiation.cc:107] Waiting for socket to connect
0925 22:44:44.429260 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0925 22:44:44.430936 (+ 1676us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:44:44.431622 (+ 686us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:44:44.431631 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:44:44.432152 (+ 521us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:44:44.432716 (+ 564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:44:44.432732 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:44:44.433656 (+ 924us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:44:44.433662 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:44:44.434656 (+ 994us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:44:44.434670 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:44:44.435285 (+ 615us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:44:44.435289 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:44:44.435513 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:44:44.436319 (+ 806us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:44:44.436347 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:44:44.440723 (+ 4376us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:44:44.443065 (+ 2342us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:44:44.443076 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:44:44.443102 (+ 26us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:44:44.443588 (+ 486us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:44:44.443993 (+ 405us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:44:44.443998 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:44:44.444003 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:44:44.444267 (+ 264us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:44:44.444789 (+ 522us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:44:44.444801 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:44:44.445238 (+ 437us) client_negotiation.cc:769] Sending connection context
0925 22:44:44.445521 (+ 283us) client_negotiation.cc:240] Negotiation successful
0925 22:44:44.446066 (+ 545us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":338,"thread_start_us":150,"threads_started":1}
I20260925 22:44:44.449361 15320 heartbeater.cc:347] Connected to a master server at 127.10.198.62:40365
I20260925 22:44:44.449710 15320 heartbeater.cc:464] Registering TS with master...
I20260925 22:44:44.450371 15320 heartbeater.cc:511] Master 127.10.198.62:40365 requested a full tablet report, sending...
I20260925 22:44:44.452447 15130 ts_manager.cc:194] Registered new tserver with Master: 117d64e10a9b4976a84373ccf4851c0e (127.10.198.1:43219)
I20260925 22:44:44.453878 15130 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.198.1@KRBTEST.COM'} at 127.10.198.1:51555
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.198.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790376259650642-11032-0/krb5kdc/kudu_127.10.198.2.keytab.
Entry for principal kudu/127.10.198.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790376259650642-11032-0/krb5kdc/kudu_127.10.198.2.keytab.
I20260925 22:44:44.460153 11032 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrWXm8X/build/debug/bin/kudu
/tmp/dist-test-taskrWXm8X/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790376259650642-11032-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790376259650642-11032-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790376259650642-11032-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790376259650642-11032-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.198.2:0
--local_ip_for_outbound_sockets=127.10.198.2
--webserver_interface=127.10.198.2
--webserver_port=0
--tserver_master_addrs=127.10.198.62:40365
--keytab_file=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790376259650642-11032-0/krb5kdc/kudu_127.10.198.2.keytab
--principal=kudu/127.10.198.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.198.20:46367
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790376259650642-11032-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790376259650642-11032-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790376259650642-11032-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:44:44.578457 15325 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:44:44.578778 15325 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:44:44.578828 15325 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:44:44.583670 15325 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:44:44.583763 15325 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:44:44.583880 15325 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.198.2
I20260925 22:44:44.590163 15325 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.198.20:46367
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790376259650642-11032-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790376259650642-11032-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790376259650642-11032-0/krb5kdc/kudu_127.10.198.2.keytab
--principal=kudu/127.10.198.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.198.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790376259650642-11032-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.198.2
--webserver_port=0
--tserver_master_addrs=127.10.198.62:40365
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15325
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.198.2
--log_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790376259650642-11032-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:09:16 UTC on ccycloud.quasar-afmhyw.root.comops.site
build id 4673379
I20260925 22:44:44.591380 15325 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:44:44.592566 15325 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:44:44.603050 15330 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:44:44.603091 15333 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:44:44.603050 15331 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:44:44.603327 15325 server_base.cc:1034] running on GCE node
I20260925 22:44:44.604238 15325 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:44:44.605052 15325 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:44:44.606276 15325 hybrid_clock.cc:648] HybridClock initialized: now 1790376284606249 us; error 54 us; skew 500 ppm
Sep 25 22:44:44 dist-test-slave-5vdr krb5kdc[15086](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376284, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.198.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:44:44.609814 15325 init.cc:377] Logged in from keytab as kudu/127.10.198.2@KRBTEST.COM (short username kudu)
I20260925 22:44:44.610746 15325 fs_manager.cc:359] Metadata directory not provided
I20260925 22:44:44.610829 15325 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:44:44.611088 15325 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:44:44.613694 15325 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790376259650642-11032-0/minicluster-data/ts-1/data/instance:
uuid: "b2dfcaefbadc4192ac7b889a1e501d14"
format_stamp: "Formatted at 2026-09-25 22:44:44 on dist-test-slave-5vdr"
server_key: "efff584794b73f9268c4be2eb6ca4ddc"
server_key_iv: "1b4ff8be4d796e8641e495f1713e0f9f"
server_key_version: "encryptionkey@0"
I20260925 22:44:44.614393 15325 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790376259650642-11032-0/minicluster-data/ts-1/wal/instance:
uuid: "b2dfcaefbadc4192ac7b889a1e501d14"
format_stamp: "Formatted at 2026-09-25 22:44:44 on dist-test-slave-5vdr"
server_key: "efff584794b73f9268c4be2eb6ca4ddc"
server_key_iv: "1b4ff8be4d796e8641e495f1713e0f9f"
server_key_version: "encryptionkey@0"
I20260925 22:44:44.618970 15325 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260925 22:44:44.622226 15339 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:44:44.623726 15325 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260925 22:44:44.623986 15325 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790376259650642-11032-0/minicluster-data/ts-1/data,/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790376259650642-11032-0/minicluster-data/ts-1/wal
uuid: "b2dfcaefbadc4192ac7b889a1e501d14"
format_stamp: "Formatted at 2026-09-25 22:44:44 on dist-test-slave-5vdr"
server_key: "efff584794b73f9268c4be2eb6ca4ddc"
server_key_iv: "1b4ff8be4d796e8641e495f1713e0f9f"
server_key_version: "encryptionkey@0"
I20260925 22:44:44.624127 15325 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790376259650642-11032-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790376259650642-11032-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790376259650642-11032-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:44:44.639966 15325 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:44:44.643424 15325 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:44:44.643648 15325 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:44:44.644357 15325 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 22:44:44.645377 15325 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:44:44.645430 15325 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:44:44.645501 15325 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:44:44.645535 15325 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 22:44:44.655260 15325 rpc_server.cc:307] RPC server started. Bound to: 127.10.198.2:41879
I20260925 22:44:44.655297 15452 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.198.2:41879 every 8 connection(s)
I20260925 22:44:44.656009 15325 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790376259650642-11032-0/minicluster-data/ts-1/data/info.pb
I20260925 22:44:44.657289 11032 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrWXm8X/build/debug/bin/kudu as pid 15325
I20260925 22:44:44.657428 11032 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790376259650642-11032-0/minicluster-data/ts-1/wal/instance
I20260925 22:44:44.657716 11032 external_mini_cluster.cc:1428] Setting key c5d5726dbe9d15b842ee94049ce067f6
WARNING: no policy specified for kudu/127.10.198.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.198.3@KRBTEST.COM" created.
Sep 25 22:44:44 dist-test-slave-5vdr krb5kdc[15086](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376284, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.198.2@KRBTEST.COM for kudu/127.10.198.62@KRBTEST.COM
I20260925 22:44:44.677537 15169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:44:44.658070 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.198.2:35227 (local address 127.10.198.62:40365)
0925 22:44:44.658227 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0925 22:44:44.658230 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 22:44:44.659459 (+ 1229us) server_negotiation.cc:407] Connection header received
0925 22:44:44.661550 (+ 2091us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:44:44.661557 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:44:44.661634 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:44:44.661746 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:44:44.663125 (+ 1379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:44:44.664112 (+ 987us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:44:44.665771 (+ 1659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:44:44.666199 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:44:44.666396 (+ 197us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:44:44.672055 (+ 5659us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:44:44.672059 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:44:44.672062 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:44:44.672094 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:44:44.673910 (+ 1816us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:44:44.674816 (+ 906us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:44:44.674823 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:44:44.674827 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:44:44.674962 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:44:44.675836 (+ 874us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:44:44.675839 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:44:44.675900 (+ 61us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:44:44.676146 (+ 246us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:44:44.676241 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0925 22:44:44.677256 (+ 1015us) server_negotiation.cc:299] Negotiation successful
0925 22:44:44.677403 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260925 22:44:44.678936 15455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:44:44.658629 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.198.62:40365 (local address 127.10.198.2:35227)
0925 22:44:44.659270 (+ 641us) negotiation.cc:107] Waiting for socket to connect
0925 22:44:44.659336 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0925 22:44:44.661246 (+ 1910us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:44:44.661941 (+ 695us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:44:44.661949 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:44:44.662360 (+ 411us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:44:44.662919 (+ 559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:44:44.662936 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:44:44.664293 (+ 1357us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:44:44.664296 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:44:44.665486 (+ 1190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:44:44.665514 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:44:44.666491 (+ 977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:44:44.666497 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:44:44.666700 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:44:44.667501 (+ 801us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:44:44.667522 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:44:44.671701 (+ 4179us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:44:44.674096 (+ 2395us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:44:44.674107 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:44:44.674132 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:44:44.674614 (+ 482us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:44:44.675089 (+ 475us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:44:44.675095 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:44:44.675099 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:44:44.675669 (+ 570us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:44:44.676400 (+ 731us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:44:44.676411 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:44:44.676874 (+ 463us) client_negotiation.cc:769] Sending connection context
0925 22:44:44.677192 (+ 318us) client_negotiation.cc:240] Negotiation successful
0925 22:44:44.677646 (+ 454us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":365,"thread_start_us":158,"threads_started":1}
I20260925 22:44:44.680794 15453 heartbeater.cc:347] Connected to a master server at 127.10.198.62:40365
I20260925 22:44:44.681097 15453 heartbeater.cc:464] Registering TS with master...
I20260925 22:44:44.681771 15453 heartbeater.cc:511] Master 127.10.198.62:40365 requested a full tablet report, sending...
I20260925 22:44:44.683267 15130 ts_manager.cc:194] Registered new tserver with Master: b2dfcaefbadc4192ac7b889a1e501d14 (127.10.198.2:41879)
I20260925 22:44:44.684161 15130 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.198.2@KRBTEST.COM'} at 127.10.198.2:35227
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.198.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790376259650642-11032-0/krb5kdc/kudu_127.10.198.3.keytab.
Entry for principal kudu/127.10.198.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790376259650642-11032-0/krb5kdc/kudu_127.10.198.3.keytab.
I20260925 22:44:44.691058 11032 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrWXm8X/build/debug/bin/kudu
/tmp/dist-test-taskrWXm8X/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790376259650642-11032-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790376259650642-11032-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790376259650642-11032-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790376259650642-11032-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.198.3:0
--local_ip_for_outbound_sockets=127.10.198.3
--webserver_interface=127.10.198.3
--webserver_port=0
--tserver_master_addrs=127.10.198.62:40365
--keytab_file=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790376259650642-11032-0/krb5kdc/kudu_127.10.198.3.keytab
--principal=kudu/127.10.198.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.198.20:46367
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790376259650642-11032-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790376259650642-11032-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790376259650642-11032-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:44:44.813382 15458 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:44:44.813704 15458 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:44:44.813753 15458 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:44:44.818141 15458 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:44:44.818229 15458 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:44:44.818298 15458 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.198.3
I20260925 22:44:44.824045 15458 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.198.20:46367
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790376259650642-11032-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790376259650642-11032-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790376259650642-11032-0/krb5kdc/kudu_127.10.198.3.keytab
--principal=kudu/127.10.198.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.198.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790376259650642-11032-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.198.3
--webserver_port=0
--tserver_master_addrs=127.10.198.62:40365
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15458
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.198.3
--log_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790376259650642-11032-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:09:16 UTC on ccycloud.quasar-afmhyw.root.comops.site
build id 4673379
I20260925 22:44:44.825198 15458 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:44:44.826249 15458 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:44:44.836912 15463 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:44:44.837021 15458 server_base.cc:1034] running on GCE node
W20260925 22:44:44.836912 15466 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:44:44.836912 15464 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:44:44.837590 15458 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:44:44.838344 15458 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:44:44.839521 15458 hybrid_clock.cc:648] HybridClock initialized: now 1790376284839500 us; error 39 us; skew 500 ppm
Sep 25 22:44:44 dist-test-slave-5vdr krb5kdc[15086](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376284, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.198.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:44:44.842716 15458 init.cc:377] Logged in from keytab as kudu/127.10.198.3@KRBTEST.COM (short username kudu)
I20260925 22:44:44.843588 15458 fs_manager.cc:359] Metadata directory not provided
I20260925 22:44:44.843703 15458 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:44:44.844012 15458 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:44:44.846534 15458 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790376259650642-11032-0/minicluster-data/ts-2/data/instance:
uuid: "f672de6f5f7e40e9be2569882b1619d7"
format_stamp: "Formatted at 2026-09-25 22:44:44 on dist-test-slave-5vdr"
server_key: "699bc34654fea4f9e975c6551e609ff0"
server_key_iv: "561f1ca4d0f398250e246c3ebbcac0ca"
server_key_version: "encryptionkey@0"
I20260925 22:44:44.847265 15458 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790376259650642-11032-0/minicluster-data/ts-2/wal/instance:
uuid: "f672de6f5f7e40e9be2569882b1619d7"
format_stamp: "Formatted at 2026-09-25 22:44:44 on dist-test-slave-5vdr"
server_key: "699bc34654fea4f9e975c6551e609ff0"
server_key_iv: "561f1ca4d0f398250e246c3ebbcac0ca"
server_key_version: "encryptionkey@0"
I20260925 22:44:44.851820 15458 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.004s
I20260925 22:44:44.854794 15472 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:44:44.856304 15458 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260925 22:44:44.856485 15458 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790376259650642-11032-0/minicluster-data/ts-2/data,/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790376259650642-11032-0/minicluster-data/ts-2/wal
uuid: "f672de6f5f7e40e9be2569882b1619d7"
format_stamp: "Formatted at 2026-09-25 22:44:44 on dist-test-slave-5vdr"
server_key: "699bc34654fea4f9e975c6551e609ff0"
server_key_iv: "561f1ca4d0f398250e246c3ebbcac0ca"
server_key_version: "encryptionkey@0"
I20260925 22:44:44.856626 15458 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790376259650642-11032-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790376259650642-11032-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790376259650642-11032-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:44:44.879815 15458 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:44:44.883733 15458 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:44:44.884003 15458 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:44:44.884656 15458 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 22:44:44.885727 15458 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:44:44.885818 15458 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:44:44.885890 15458 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:44:44.885924 15458 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 22:44:44.896322 15458 rpc_server.cc:307] RPC server started. Bound to: 127.10.198.3:46589
I20260925 22:44:44.896348 15585 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.198.3:46589 every 8 connection(s)
I20260925 22:44:44.897040 15458 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790376259650642-11032-0/minicluster-data/ts-2/data/info.pb
I20260925 22:44:44.898804 11032 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrWXm8X/build/debug/bin/kudu as pid 15458
I20260925 22:44:44.898952 11032 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790376259650642-11032-0/minicluster-data/ts-2/wal/instance
I20260925 22:44:44.899264 11032 external_mini_cluster.cc:1428] Setting key 43b1e96c7ed48ed3c35fec7f344ab5da
Sep 25 22:44:44 dist-test-slave-5vdr krb5kdc[15086](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376284, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.198.3@KRBTEST.COM for kudu/127.10.198.62@KRBTEST.COM
I20260925 22:44:44.913085 15169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:44:44.899073 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.198.3:53711 (local address 127.10.198.62:40365)
0925 22:44:44.899219 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0925 22:44:44.899224 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:44:44.900496 (+ 1272us) server_negotiation.cc:407] Connection header received
0925 22:44:44.902233 (+ 1737us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:44:44.902236 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:44:44.902298 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:44:44.902373 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:44:44.903749 (+ 1376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:44:44.904387 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:44:44.905737 (+ 1350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:44:44.905990 (+ 253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:44:44.906119 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:44:44.909140 (+ 3021us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:44:44.909144 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:44:44.909146 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:44:44.909178 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:44:44.910561 (+ 1383us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:44:44.911239 (+ 678us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:44:44.911242 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:44:44.911243 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:44:44.911327 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:44:44.911654 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:44:44.911656 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:44:44.911657 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:44:44.911911 (+ 254us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:44:44.912014 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0925 22:44:44.912737 (+ 723us) server_negotiation.cc:299] Negotiation successful
0925 22:44:44.912897 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260925 22:44:44.914204 15588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:44:44.899644 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.198.62:40365 (local address 127.10.198.3:53711)
0925 22:44:44.900310 (+ 666us) negotiation.cc:107] Waiting for socket to connect
0925 22:44:44.900377 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0925 22:44:44.901996 (+ 1619us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:44:44.902576 (+ 580us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:44:44.902585 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:44:44.903037 (+ 452us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:44:44.903579 (+ 542us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:44:44.903594 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:44:44.904549 (+ 955us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:44:44.904553 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:44:44.905563 (+ 1010us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:44:44.905575 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:44:44.906178 (+ 603us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:44:44.906183 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:44:44.906389 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:44:44.907163 (+ 774us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:44:44.907181 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:44:44.908914 (+ 1733us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:44:44.910713 (+ 1799us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:44:44.910719 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:44:44.910737 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:44:44.911136 (+ 399us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:44:44.911414 (+ 278us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:44:44.911416 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:44:44.911417 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:44:44.911570 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:44:44.912033 (+ 463us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:44:44.912040 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:44:44.912407 (+ 367us) client_negotiation.cc:769] Sending connection context
0925 22:44:44.912686 (+ 279us) client_negotiation.cc:240] Negotiation successful
0925 22:44:44.913062 (+ 376us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":402,"thread_start_us":198,"threads_started":1}
I20260925 22:44:44.915524 15586 heartbeater.cc:347] Connected to a master server at 127.10.198.62:40365
I20260925 22:44:44.915771 15586 heartbeater.cc:464] Registering TS with master...
I20260925 22:44:44.916358 15586 heartbeater.cc:511] Master 127.10.198.62:40365 requested a full tablet report, sending...
I20260925 22:44:44.917532 15130 ts_manager.cc:194] Registered new tserver with Master: f672de6f5f7e40e9be2569882b1619d7 (127.10.198.3:46589)
I20260925 22:44:44.918177 15130 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.198.3@KRBTEST.COM'} at 127.10.198.3:53711
I20260925 22:44:44.926177 11032 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 22:44:44.936571 15169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:44:44.928473 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55318 (local address 127.10.198.62:40365)
0925 22:44:44.928648 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0925 22:44:44.928653 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:44:44.928746 (+ 93us) server_negotiation.cc:407] Connection header received
0925 22:44:44.928930 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:44:44.928934 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:44:44.928991 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:44:44.929086 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:44:44.929867 (+ 781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:44:44.930408 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:44:44.931286 (+ 878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:44:44.931520 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:44:44.931638 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:44:44.933173 (+ 1535us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:44:44.933178 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:44:44.933180 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:44:44.933220 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:44:44.934634 (+ 1414us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:44:44.935142 (+ 508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:44:44.935146 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:44:44.935148 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:44:44.935243 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:44:44.935555 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:44:44.935559 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:44:44.935560 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:44:44.935822 (+ 262us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:44:44.936013 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0925 22:44:44.936262 (+ 249us) server_negotiation.cc:299] Negotiation successful
0925 22:44:44.936394 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260925 22:44:44.939672 15130 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55318:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 22:44:44.942852 15130 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 22:44:44.959034 15602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:44:44.953627 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59162 (local address 127.10.198.2:41879)
0925 22:44:44.954191 (+ 564us) server_negotiation.cc:206] Beginning negotiation
0925 22:44:44.954202 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 22:44:44.954221 (+ 19us) server_negotiation.cc:407] Connection header received
0925 22:44:44.954284 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:44:44.954288 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:44:44.954429 (+ 141us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:44:44.954571 (+ 142us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:44:44.955050 (+ 479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:44:44.956150 (+ 1100us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:44:44.957607 (+ 1457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:44:44.958207 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:44:44.958350 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:44:44.958520 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0925 22:44:44.958727 (+ 207us) server_negotiation.cc:299] Negotiation successful
0925 22:44:44.958851 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":473,"thread_start_us":110,"threads_started":1}
I20260925 22:44:44.959044 15597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:44:44.952413 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.198.3:46589 (local address 127.0.0.1:46668)
0925 22:44:44.952962 (+ 549us) negotiation.cc:107] Waiting for socket to connect
0925 22:44:44.953015 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0925 22:44:44.953156 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:44:44.954027 (+ 871us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:44:44.954037 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:44:44.954064 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:44:44.954236 (+ 172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:44:44.954241 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:44:44.955875 (+ 1634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:44:44.955878 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:44:44.957413 (+ 1535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:44:44.957425 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:44:44.958484 (+ 1059us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:44:44.958488 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:44:44.958580 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:44:44.958602 (+ 22us) client_negotiation.cc:769] Sending connection context
0925 22:44:44.958656 (+ 54us) client_negotiation.cc:240] Negotiation successful
0925 22:44:44.958867 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":379,"mutex_wait_us":196,"thread_start_us":97,"threads_started":1}
I20260925 22:44:44.959029 15600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:44:44.953217 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.198.2:41879 (local address 127.0.0.1:59162)
0925 22:44:44.954004 (+ 787us) negotiation.cc:107] Waiting for socket to connect
0925 22:44:44.954017 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 22:44:44.954085 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:44:44.954783 (+ 698us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:44:44.954787 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:44:44.954809 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:44:44.954914 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:44:44.954921 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:44:44.956304 (+ 1383us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:44:44.956308 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:44:44.957464 (+ 1156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:44:44.957474 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:44:44.958364 (+ 890us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:44:44.958368 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:44:44.958484 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:44:44.958538 (+ 54us) client_negotiation.cc:769] Sending connection context
0925 22:44:44.958691 (+ 153us) client_negotiation.cc:240] Negotiation successful
0925 22:44:44.958851 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":715,"mutex_wait_us":8,"thread_start_us":96,"threads_started":1}
I20260925 22:44:44.959287 15601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:44:44.953277 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34072 (local address 127.10.198.1:43219)
0925 22:44:44.954091 (+ 814us) server_negotiation.cc:206] Beginning negotiation
0925 22:44:44.954098 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 22:44:44.954122 (+ 24us) server_negotiation.cc:407] Connection header received
0925 22:44:44.954197 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:44:44.954204 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:44:44.954415 (+ 211us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:44:44.954571 (+ 156us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:44:44.954817 (+ 246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:44:44.955625 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:44:44.957433 (+ 1808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:44:44.958329 (+ 896us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:44:44.958458 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:44:44.958604 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0925 22:44:44.959008 (+ 404us) server_negotiation.cc:299] Negotiation successful
0925 22:44:44.959116 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":706,"thread_start_us":86,"threads_started":1}
I20260925 22:44:44.960153 15598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:44:44.952615 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.198.1:43219 (local address 127.0.0.1:34072)
0925 22:44:44.953620 (+ 1005us) negotiation.cc:107] Waiting for socket to connect
0925 22:44:44.953636 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 22:44:44.953728 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:44:44.954571 (+ 843us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:44:44.954574 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:44:44.954592 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:44:44.954683 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:44:44.954689 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:44:44.955759 (+ 1070us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:44:44.955764 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:44:44.957281 (+ 1517us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:44:44.957300 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:44:44.958665 (+ 1365us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:44:44.958668 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:44:44.958755 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:44:44.958779 (+ 24us) client_negotiation.cc:769] Sending connection context
0925 22:44:44.958859 (+ 80us) client_negotiation.cc:240] Negotiation successful
0925 22:44:44.958916 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":939,"mutex_wait_us":118,"thread_start_us":95,"threads_started":1}
I20260925 22:44:44.960605 15599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:44:44.952602 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46668 (local address 127.10.198.3:46589)
0925 22:44:44.953473 (+ 871us) server_negotiation.cc:206] Beginning negotiation
0925 22:44:44.953492 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0925 22:44:44.953517 (+ 25us) server_negotiation.cc:407] Connection header received
0925 22:44:44.953598 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:44:44.953604 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:44:44.953834 (+ 230us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:44:44.954376 (+ 542us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:44:44.954531 (+ 155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:44:44.955702 (+ 1171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:44:44.957555 (+ 1853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:44:44.958341 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:44:44.959090 (+ 749us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:44:44.959779 (+ 689us) server_negotiation.cc:1035] Waiting for connection context
0925 22:44:44.959893 (+ 114us) server_negotiation.cc:299] Negotiation successful
0925 22:44:44.960457 (+ 564us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":762,"thread_start_us":172,"threads_started":1}
I20260925 22:44:44.962800 15387 tablet_service.cc:1467] Processing CreateTablet for tablet 1ffd07f6fcda4ff5899b6c547dcabff9 (DEFAULT_TABLE table=test-table [id=9b0f6e7ce5644bb8bab82c895fc3129c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 22:44:44.963322 15254 tablet_service.cc:1467] Processing CreateTablet for tablet 1ffd07f6fcda4ff5899b6c547dcabff9 (DEFAULT_TABLE table=test-table [id=9b0f6e7ce5644bb8bab82c895fc3129c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 22:44:44.964177 15387 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1ffd07f6fcda4ff5899b6c547dcabff9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:44:44.964421 15254 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1ffd07f6fcda4ff5899b6c547dcabff9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:44:44.965873 15520 tablet_service.cc:1467] Processing CreateTablet for tablet 1ffd07f6fcda4ff5899b6c547dcabff9 (DEFAULT_TABLE table=test-table [id=9b0f6e7ce5644bb8bab82c895fc3129c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 22:44:44.966864 15520 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1ffd07f6fcda4ff5899b6c547dcabff9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:44:44.972244 15603 tablet_bootstrap.cc:492] T 1ffd07f6fcda4ff5899b6c547dcabff9 P 117d64e10a9b4976a84373ccf4851c0e: Bootstrap starting.
I20260925 22:44:44.973188 15604 tablet_bootstrap.cc:492] T 1ffd07f6fcda4ff5899b6c547dcabff9 P b2dfcaefbadc4192ac7b889a1e501d14: Bootstrap starting.
I20260925 22:44:44.973739 15605 tablet_bootstrap.cc:492] T 1ffd07f6fcda4ff5899b6c547dcabff9 P f672de6f5f7e40e9be2569882b1619d7: Bootstrap starting.
I20260925 22:44:44.975199 15603 tablet_bootstrap.cc:654] T 1ffd07f6fcda4ff5899b6c547dcabff9 P 117d64e10a9b4976a84373ccf4851c0e: Neither blocks nor log segments found. Creating new log.
I20260925 22:44:44.976029 15604 tablet_bootstrap.cc:654] T 1ffd07f6fcda4ff5899b6c547dcabff9 P b2dfcaefbadc4192ac7b889a1e501d14: Neither blocks nor log segments found. Creating new log.
I20260925 22:44:44.976186 15603 log.cc:824] T 1ffd07f6fcda4ff5899b6c547dcabff9 P 117d64e10a9b4976a84373ccf4851c0e: Log is configured to *not* fsync() on all Append() calls
I20260925 22:44:44.976280 15605 tablet_bootstrap.cc:654] T 1ffd07f6fcda4ff5899b6c547dcabff9 P f672de6f5f7e40e9be2569882b1619d7: Neither blocks nor log segments found. Creating new log.
I20260925 22:44:44.977011 15604 log.cc:824] T 1ffd07f6fcda4ff5899b6c547dcabff9 P b2dfcaefbadc4192ac7b889a1e501d14: Log is configured to *not* fsync() on all Append() calls
I20260925 22:44:44.977566 15605 log.cc:824] T 1ffd07f6fcda4ff5899b6c547dcabff9 P f672de6f5f7e40e9be2569882b1619d7: Log is configured to *not* fsync() on all Append() calls
I20260925 22:44:44.978596 15603 tablet_bootstrap.cc:492] T 1ffd07f6fcda4ff5899b6c547dcabff9 P 117d64e10a9b4976a84373ccf4851c0e: No bootstrap required, opened a new log
I20260925 22:44:44.978839 15603 ts_tablet_manager.cc:1397] T 1ffd07f6fcda4ff5899b6c547dcabff9 P 117d64e10a9b4976a84373ccf4851c0e: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260925 22:44:44.979135 15604 tablet_bootstrap.cc:492] T 1ffd07f6fcda4ff5899b6c547dcabff9 P b2dfcaefbadc4192ac7b889a1e501d14: No bootstrap required, opened a new log
I20260925 22:44:44.979351 15604 ts_tablet_manager.cc:1397] T 1ffd07f6fcda4ff5899b6c547dcabff9 P b2dfcaefbadc4192ac7b889a1e501d14: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.004s
I20260925 22:44:44.979651 15605 tablet_bootstrap.cc:492] T 1ffd07f6fcda4ff5899b6c547dcabff9 P f672de6f5f7e40e9be2569882b1619d7: No bootstrap required, opened a new log
I20260925 22:44:44.979875 15605 ts_tablet_manager.cc:1397] T 1ffd07f6fcda4ff5899b6c547dcabff9 P f672de6f5f7e40e9be2569882b1619d7: Time spent bootstrapping tablet: real 0.006s user 0.006s sys 0.000s
I20260925 22:44:44.983163 15603 raft_consensus.cc:348] T 1ffd07f6fcda4ff5899b6c547dcabff9 P 117d64e10a9b4976a84373ccf4851c0e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f672de6f5f7e40e9be2569882b1619d7" member_type: VOTER last_known_addr { host: "127.10.198.3" port: 46589 } } peers { permanent_uuid: "117d64e10a9b4976a84373ccf4851c0e" member_type: VOTER last_known_addr { host: "127.10.198.1" port: 43219 } } peers { permanent_uuid: "b2dfcaefbadc4192ac7b889a1e501d14" member_type: VOTER last_known_addr { host: "127.10.198.2" port: 41879 } }
I20260925 22:44:44.983543 15603 raft_consensus.cc:374] T 1ffd07f6fcda4ff5899b6c547dcabff9 P 117d64e10a9b4976a84373ccf4851c0e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:44:44.983604 15603 raft_consensus.cc:729] T 1ffd07f6fcda4ff5899b6c547dcabff9 P 117d64e10a9b4976a84373ccf4851c0e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 117d64e10a9b4976a84373ccf4851c0e, State: Initialized, Role: FOLLOWER
I20260925 22:44:44.983548 15604 raft_consensus.cc:348] T 1ffd07f6fcda4ff5899b6c547dcabff9 P b2dfcaefbadc4192ac7b889a1e501d14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f672de6f5f7e40e9be2569882b1619d7" member_type: VOTER last_known_addr { host: "127.10.198.3" port: 46589 } } peers { permanent_uuid: "117d64e10a9b4976a84373ccf4851c0e" member_type: VOTER last_known_addr { host: "127.10.198.1" port: 43219 } } peers { permanent_uuid: "b2dfcaefbadc4192ac7b889a1e501d14" member_type: VOTER last_known_addr { host: "127.10.198.2" port: 41879 } }
I20260925 22:44:44.983884 15604 raft_consensus.cc:374] T 1ffd07f6fcda4ff5899b6c547dcabff9 P b2dfcaefbadc4192ac7b889a1e501d14 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:44:44.983947 15604 raft_consensus.cc:729] T 1ffd07f6fcda4ff5899b6c547dcabff9 P b2dfcaefbadc4192ac7b889a1e501d14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b2dfcaefbadc4192ac7b889a1e501d14, State: Initialized, Role: FOLLOWER
I20260925 22:44:44.983976 15605 raft_consensus.cc:348] T 1ffd07f6fcda4ff5899b6c547dcabff9 P f672de6f5f7e40e9be2569882b1619d7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f672de6f5f7e40e9be2569882b1619d7" member_type: VOTER last_known_addr { host: "127.10.198.3" port: 46589 } } peers { permanent_uuid: "117d64e10a9b4976a84373ccf4851c0e" member_type: VOTER last_known_addr { host: "127.10.198.1" port: 43219 } } peers { permanent_uuid: "b2dfcaefbadc4192ac7b889a1e501d14" member_type: VOTER last_known_addr { host: "127.10.198.2" port: 41879 } }
I20260925 22:44:44.984241 15605 raft_consensus.cc:374] T 1ffd07f6fcda4ff5899b6c547dcabff9 P f672de6f5f7e40e9be2569882b1619d7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:44:44.984292 15605 raft_consensus.cc:729] T 1ffd07f6fcda4ff5899b6c547dcabff9 P f672de6f5f7e40e9be2569882b1619d7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f672de6f5f7e40e9be2569882b1619d7, State: Initialized, Role: FOLLOWER
I20260925 22:44:44.984236 15603 consensus_queue.cc:260] T 1ffd07f6fcda4ff5899b6c547dcabff9 P 117d64e10a9b4976a84373ccf4851c0e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f672de6f5f7e40e9be2569882b1619d7" member_type: VOTER last_known_addr { host: "127.10.198.3" port: 46589 } } peers { permanent_uuid: "117d64e10a9b4976a84373ccf4851c0e" member_type: VOTER last_known_addr { host: "127.10.198.1" port: 43219 } } peers { permanent_uuid: "b2dfcaefbadc4192ac7b889a1e501d14" member_type: VOTER last_known_addr { host: "127.10.198.2" port: 41879 } }
I20260925 22:44:44.984449 15604 consensus_queue.cc:260] T 1ffd07f6fcda4ff5899b6c547dcabff9 P b2dfcaefbadc4192ac7b889a1e501d14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f672de6f5f7e40e9be2569882b1619d7" member_type: VOTER last_known_addr { host: "127.10.198.3" port: 46589 } } peers { permanent_uuid: "117d64e10a9b4976a84373ccf4851c0e" member_type: VOTER last_known_addr { host: "127.10.198.1" port: 43219 } } peers { permanent_uuid: "b2dfcaefbadc4192ac7b889a1e501d14" member_type: VOTER last_known_addr { host: "127.10.198.2" port: 41879 } }
I20260925 22:44:44.984742 15605 consensus_queue.cc:260] T 1ffd07f6fcda4ff5899b6c547dcabff9 P f672de6f5f7e40e9be2569882b1619d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f672de6f5f7e40e9be2569882b1619d7" member_type: VOTER last_known_addr { host: "127.10.198.3" port: 46589 } } peers { permanent_uuid: "117d64e10a9b4976a84373ccf4851c0e" member_type: VOTER last_known_addr { host: "127.10.198.1" port: 43219 } } peers { permanent_uuid: "b2dfcaefbadc4192ac7b889a1e501d14" member_type: VOTER last_known_addr { host: "127.10.198.2" port: 41879 } }
I20260925 22:44:44.985441 15586 heartbeater.cc:503] Master 127.10.198.62:40365 was elected leader, sending a full tablet report...
I20260925 22:44:44.985450 15603 ts_tablet_manager.cc:1428] T 1ffd07f6fcda4ff5899b6c547dcabff9 P 117d64e10a9b4976a84373ccf4851c0e: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260925 22:44:44.985582 15605 ts_tablet_manager.cc:1428] T 1ffd07f6fcda4ff5899b6c547dcabff9 P f672de6f5f7e40e9be2569882b1619d7: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260925 22:44:44.985829 15320 heartbeater.cc:503] Master 127.10.198.62:40365 was elected leader, sending a full tablet report...
I20260925 22:44:44.986620 15604 ts_tablet_manager.cc:1428] T 1ffd07f6fcda4ff5899b6c547dcabff9 P b2dfcaefbadc4192ac7b889a1e501d14: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260925 22:44:44.986979 15453 heartbeater.cc:503] Master 127.10.198.62:40365 was elected leader, sending a full tablet report...
I20260925 22:44:45.091043 15609 raft_consensus.cc:482] T 1ffd07f6fcda4ff5899b6c547dcabff9 P 117d64e10a9b4976a84373ccf4851c0e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 22:44:45.091323 15609 raft_consensus.cc:504] T 1ffd07f6fcda4ff5899b6c547dcabff9 P 117d64e10a9b4976a84373ccf4851c0e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f672de6f5f7e40e9be2569882b1619d7" member_type: VOTER last_known_addr { host: "127.10.198.3" port: 46589 } } peers { permanent_uuid: "117d64e10a9b4976a84373ccf4851c0e" member_type: VOTER last_known_addr { host: "127.10.198.1" port: 43219 } } peers { permanent_uuid: "b2dfcaefbadc4192ac7b889a1e501d14" member_type: VOTER last_known_addr { host: "127.10.198.2" port: 41879 } }
I20260925 22:44:45.092813 15609 leader_election.cc:288] T 1ffd07f6fcda4ff5899b6c547dcabff9 P 117d64e10a9b4976a84373ccf4851c0e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f672de6f5f7e40e9be2569882b1619d7 (127.10.198.3:46589), b2dfcaefbadc4192ac7b889a1e501d14 (127.10.198.2:41879)
I20260925 22:44:45.097704 15612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:44:45.093185 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.198.3:46589 (local address 127.10.198.1:44759)
0925 22:44:45.093612 (+ 427us) negotiation.cc:107] Waiting for socket to connect
0925 22:44:45.093629 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 22:44:45.093752 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:44:45.094330 (+ 578us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:44:45.094333 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:44:45.094355 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:44:45.094516 (+ 161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:44:45.094523 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:44:45.095367 (+ 844us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:44:45.095371 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:44:45.096357 (+ 986us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:44:45.096368 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:44:45.097177 (+ 809us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:44:45.097181 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:44:45.097314 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:44:45.097343 (+ 29us) client_negotiation.cc:769] Sending connection context
0925 22:44:45.097430 (+ 87us) client_negotiation.cc:240] Negotiation successful
0925 22:44:45.097502 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":345,"thread_start_us":235,"threads_started":1}
I20260925 22:44:45.097708 15599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:44:45.093313 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.198.1:44759 (local address 127.10.198.3:46589)
0925 22:44:45.093456 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0925 22:44:45.093460 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:44:45.093729 (+ 269us) server_negotiation.cc:407] Connection header received
0925 22:44:45.094066 (+ 337us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:44:45.094069 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:44:45.094124 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:44:45.094239 (+ 115us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:44:45.094656 (+ 417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:44:45.095230 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:44:45.096513 (+ 1283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:44:45.097021 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:44:45.097137 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:44:45.097182 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0925 22:44:45.097470 (+ 288us) server_negotiation.cc:299] Negotiation successful
0925 22:44:45.097571 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260925 22:44:45.097851 15602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:44:45.093922 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.198.1:51935 (local address 127.10.198.2:41879)
0925 22:44:45.094104 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0925 22:44:45.094108 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:44:45.094281 (+ 173us) server_negotiation.cc:407] Connection header received
0925 22:44:45.094339 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:44:45.094341 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:44:45.094389 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:44:45.094482 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:44:45.094865 (+ 383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:44:45.095439 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:44:45.096693 (+ 1254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:44:45.097184 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:44:45.097315 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:44:45.097364 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0925 22:44:45.097623 (+ 259us) server_negotiation.cc:299] Negotiation successful
0925 22:44:45.097703 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260925 22:44:45.098131 15613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:44:45.093797 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.198.2:41879 (local address 127.10.198.1:51935)
0925 22:44:45.094130 (+ 333us) negotiation.cc:107] Waiting for socket to connect
0925 22:44:45.094145 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 22:44:45.094249 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:44:45.094518 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:44:45.094520 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:44:45.094536 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:44:45.094694 (+ 158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:44:45.094701 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:44:45.095576 (+ 875us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:44:45.095579 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:44:45.096548 (+ 969us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:44:45.096559 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:44:45.097321 (+ 762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:44:45.097325 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:44:45.097476 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:44:45.097501 (+ 25us) client_negotiation.cc:769] Sending connection context
0925 22:44:45.097583 (+ 82us) client_negotiation.cc:240] Negotiation successful
0925 22:44:45.097657 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":263,"thread_start_us":105,"threads_started":1}
I20260925 22:44:45.098614 15540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ffd07f6fcda4ff5899b6c547dcabff9" candidate_uuid: "117d64e10a9b4976a84373ccf4851c0e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f672de6f5f7e40e9be2569882b1619d7" is_pre_election: true
I20260925 22:44:45.098798 15407 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ffd07f6fcda4ff5899b6c547dcabff9" candidate_uuid: "117d64e10a9b4976a84373ccf4851c0e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b2dfcaefbadc4192ac7b889a1e501d14" is_pre_election: true
I20260925 22:44:45.099014 15540 raft_consensus.cc:2454] T 1ffd07f6fcda4ff5899b6c547dcabff9 P f672de6f5f7e40e9be2569882b1619d7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 117d64e10a9b4976a84373ccf4851c0e in term 0.
I20260925 22:44:45.099125 15407 raft_consensus.cc:2454] T 1ffd07f6fcda4ff5899b6c547dcabff9 P b2dfcaefbadc4192ac7b889a1e501d14 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 117d64e10a9b4976a84373ccf4851c0e in term 0.
I20260925 22:44:45.099603 15209 leader_election.cc:302] T 1ffd07f6fcda4ff5899b6c547dcabff9 P 117d64e10a9b4976a84373ccf4851c0e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 117d64e10a9b4976a84373ccf4851c0e, f672de6f5f7e40e9be2569882b1619d7; no voters:
I20260925 22:44:45.099938 15609 raft_consensus.cc:2781] T 1ffd07f6fcda4ff5899b6c547dcabff9 P 117d64e10a9b4976a84373ccf4851c0e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 22:44:45.100046 15609 raft_consensus.cc:482] T 1ffd07f6fcda4ff5899b6c547dcabff9 P 117d64e10a9b4976a84373ccf4851c0e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 22:44:45.100092 15609 raft_consensus.cc:3037] T 1ffd07f6fcda4ff5899b6c547dcabff9 P 117d64e10a9b4976a84373ccf4851c0e [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:44:45.101186 15609 raft_consensus.cc:504] T 1ffd07f6fcda4ff5899b6c547dcabff9 P 117d64e10a9b4976a84373ccf4851c0e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f672de6f5f7e40e9be2569882b1619d7" member_type: VOTER last_known_addr { host: "127.10.198.3" port: 46589 } } peers { permanent_uuid: "117d64e10a9b4976a84373ccf4851c0e" member_type: VOTER last_known_addr { host: "127.10.198.1" port: 43219 } } peers { permanent_uuid: "b2dfcaefbadc4192ac7b889a1e501d14" member_type: VOTER last_known_addr { host: "127.10.198.2" port: 41879 } }
I20260925 22:44:45.101608 15609 leader_election.cc:288] T 1ffd07f6fcda4ff5899b6c547dcabff9 P 117d64e10a9b4976a84373ccf4851c0e [CANDIDATE]: Term 1 election: Requested vote from peers f672de6f5f7e40e9be2569882b1619d7 (127.10.198.3:46589), b2dfcaefbadc4192ac7b889a1e501d14 (127.10.198.2:41879)
I20260925 22:44:45.102136 15540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ffd07f6fcda4ff5899b6c547dcabff9" candidate_uuid: "117d64e10a9b4976a84373ccf4851c0e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f672de6f5f7e40e9be2569882b1619d7"
I20260925 22:44:45.102190 15407 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ffd07f6fcda4ff5899b6c547dcabff9" candidate_uuid: "117d64e10a9b4976a84373ccf4851c0e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b2dfcaefbadc4192ac7b889a1e501d14"
I20260925 22:44:45.102301 15540 raft_consensus.cc:3037] T 1ffd07f6fcda4ff5899b6c547dcabff9 P f672de6f5f7e40e9be2569882b1619d7 [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:44:45.102357 15407 raft_consensus.cc:3037] T 1ffd07f6fcda4ff5899b6c547dcabff9 P b2dfcaefbadc4192ac7b889a1e501d14 [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:44:45.103531 15540 raft_consensus.cc:2454] T 1ffd07f6fcda4ff5899b6c547dcabff9 P f672de6f5f7e40e9be2569882b1619d7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 117d64e10a9b4976a84373ccf4851c0e in term 1.
I20260925 22:44:45.103531 15407 raft_consensus.cc:2454] T 1ffd07f6fcda4ff5899b6c547dcabff9 P b2dfcaefbadc4192ac7b889a1e501d14 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 117d64e10a9b4976a84373ccf4851c0e in term 1.
I20260925 22:44:45.104004 15209 leader_election.cc:302] T 1ffd07f6fcda4ff5899b6c547dcabff9 P 117d64e10a9b4976a84373ccf4851c0e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 117d64e10a9b4976a84373ccf4851c0e, f672de6f5f7e40e9be2569882b1619d7; no voters:
I20260925 22:44:45.104223 15609 raft_consensus.cc:2781] T 1ffd07f6fcda4ff5899b6c547dcabff9 P 117d64e10a9b4976a84373ccf4851c0e [term 1 FOLLOWER]: Leader election won for term 1
I20260925 22:44:45.104481 15609 raft_consensus.cc:686] T 1ffd07f6fcda4ff5899b6c547dcabff9 P 117d64e10a9b4976a84373ccf4851c0e [term 1 LEADER]: Becoming Leader. State: Replica: 117d64e10a9b4976a84373ccf4851c0e, State: Running, Role: LEADER
I20260925 22:44:45.104892 15609 consensus_queue.cc:237] T 1ffd07f6fcda4ff5899b6c547dcabff9 P 117d64e10a9b4976a84373ccf4851c0e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f672de6f5f7e40e9be2569882b1619d7" member_type: VOTER last_known_addr { host: "127.10.198.3" port: 46589 } } peers { permanent_uuid: "117d64e10a9b4976a84373ccf4851c0e" member_type: VOTER last_known_addr { host: "127.10.198.1" port: 43219 } } peers { permanent_uuid: "b2dfcaefbadc4192ac7b889a1e501d14" member_type: VOTER last_known_addr { host: "127.10.198.2" port: 41879 } }
I20260925 22:44:45.108767 15129 catalog_manager.cc:5495] T 1ffd07f6fcda4ff5899b6c547dcabff9 P 117d64e10a9b4976a84373ccf4851c0e reported cstate change: term changed from 0 to 1, leader changed from <none> to 117d64e10a9b4976a84373ccf4851c0e (127.10.198.1). New cstate: current_term: 1 leader_uuid: "117d64e10a9b4976a84373ccf4851c0e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f672de6f5f7e40e9be2569882b1619d7" member_type: VOTER last_known_addr { host: "127.10.198.3" port: 46589 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "117d64e10a9b4976a84373ccf4851c0e" member_type: VOTER last_known_addr { host: "127.10.198.1" port: 43219 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b2dfcaefbadc4192ac7b889a1e501d14" member_type: VOTER last_known_addr { host: "127.10.198.2" port: 41879 } health_report { overall_health: UNKNOWN } } }
W20260925 22:44:45.148252 15587 tablet.cc:2369] T 1ffd07f6fcda4ff5899b6c547dcabff9 P f672de6f5f7e40e9be2569882b1619d7: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 22:44:45.154323 15601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:44:45.150293 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34084 (local address 127.10.198.1:43219)
0925 22:44:45.150475 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0925 22:44:45.150481 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 22:44:45.150500 (+ 19us) server_negotiation.cc:407] Connection header received
0925 22:44:45.150748 (+ 248us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:44:45.150751 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:44:45.150810 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:44:45.150928 (+ 118us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 22:44:45.151223 (+ 295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:44:45.151740 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:44:45.152695 (+ 955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:44:45.152927 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:44:45.153044 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:44:45.153406 (+ 362us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 22:44:45.153885 (+ 479us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 22:44:45.154058 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0925 22:44:45.154092 (+ 34us) server_negotiation.cc:299] Negotiation successful
0925 22:44:45.154162 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
W20260925 22:44:45.157202 15454 tablet.cc:2369] T 1ffd07f6fcda4ff5899b6c547dcabff9 P b2dfcaefbadc4192ac7b889a1e501d14: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 22:44:45.161546 15540 raft_consensus.cc:1261] T 1ffd07f6fcda4ff5899b6c547dcabff9 P f672de6f5f7e40e9be2569882b1619d7 [term 1 FOLLOWER]: Refusing update from remote peer 117d64e10a9b4976a84373ccf4851c0e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 22:44:45.161549 15407 raft_consensus.cc:1261] T 1ffd07f6fcda4ff5899b6c547dcabff9 P b2dfcaefbadc4192ac7b889a1e501d14 [term 1 FOLLOWER]: Refusing update from remote peer 117d64e10a9b4976a84373ccf4851c0e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 22:44:45.162398 15614 consensus_queue.cc:1035] T 1ffd07f6fcda4ff5899b6c547dcabff9 P 117d64e10a9b4976a84373ccf4851c0e [LEADER]: Connected to new peer: Peer: permanent_uuid: "b2dfcaefbadc4192ac7b889a1e501d14" member_type: VOTER last_known_addr { host: "127.10.198.2" port: 41879 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 22:44:45.162619 15609 consensus_queue.cc:1035] T 1ffd07f6fcda4ff5899b6c547dcabff9 P 117d64e10a9b4976a84373ccf4851c0e [LEADER]: Connected to new peer: Peer: permanent_uuid: "f672de6f5f7e40e9be2569882b1619d7" member_type: VOTER last_known_addr { host: "127.10.198.3" port: 46589 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 22:44:45.181965 15130 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55318:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 22:44:45.182217 15130 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55318:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 22:44:45.185196 15130 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 87b1e94f8c0f4aa0b240b08515856297: Sending DeleteTablet for 3 replicas of tablet 1ffd07f6fcda4ff5899b6c547dcabff9
I20260925 22:44:45.186216 15254 tablet_service.cc:1514] Processing DeleteTablet for tablet 1ffd07f6fcda4ff5899b6c547dcabff9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 22:44:45 UTC) from {username='kudu', principal='kudu/127.10.198.62@KRBTEST.COM'} at 127.0.0.1:34072
I20260925 22:44:45.186237 15387 tablet_service.cc:1514] Processing DeleteTablet for tablet 1ffd07f6fcda4ff5899b6c547dcabff9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 22:44:45 UTC) from {username='kudu', principal='kudu/127.10.198.62@KRBTEST.COM'} at 127.0.0.1:59162
I20260925 22:44:45.186265 15520 tablet_service.cc:1514] Processing DeleteTablet for tablet 1ffd07f6fcda4ff5899b6c547dcabff9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 22:44:45 UTC) from {username='kudu', principal='kudu/127.10.198.62@KRBTEST.COM'} at 127.0.0.1:46668
I20260925 22:44:45.186827 15629 tablet_replica.cc:330] T 1ffd07f6fcda4ff5899b6c547dcabff9 P f672de6f5f7e40e9be2569882b1619d7: stopping tablet replica
I20260925 22:44:45.187058 11032 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrWXm8X/build/debug/bin/kudu with pid 15192
I20260925 22:44:45.187230 15629 raft_consensus.cc:2229] T 1ffd07f6fcda4ff5899b6c547dcabff9 P f672de6f5f7e40e9be2569882b1619d7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 22:44:45.187716 15629 raft_consensus.cc:2258] T 1ffd07f6fcda4ff5899b6c547dcabff9 P f672de6f5f7e40e9be2569882b1619d7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 22:44:45.187829 15627 tablet_replica.cc:330] T 1ffd07f6fcda4ff5899b6c547dcabff9 P b2dfcaefbadc4192ac7b889a1e501d14: stopping tablet replica
I20260925 22:44:45.188644 15627 raft_consensus.cc:2229] T 1ffd07f6fcda4ff5899b6c547dcabff9 P b2dfcaefbadc4192ac7b889a1e501d14 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 22:44:45.188997 15627 raft_consensus.cc:2258] T 1ffd07f6fcda4ff5899b6c547dcabff9 P b2dfcaefbadc4192ac7b889a1e501d14 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 22:44:45.189409 15629 ts_tablet_manager.cc:1905] T 1ffd07f6fcda4ff5899b6c547dcabff9 P f672de6f5f7e40e9be2569882b1619d7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 22:44:45.190922 15627 ts_tablet_manager.cc:1905] T 1ffd07f6fcda4ff5899b6c547dcabff9 P b2dfcaefbadc4192ac7b889a1e501d14: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 22:44:45.193075 15629 ts_tablet_manager.cc:1918] T 1ffd07f6fcda4ff5899b6c547dcabff9 P f672de6f5f7e40e9be2569882b1619d7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 22:44:45.193212 15629 log.cc:1196] T 1ffd07f6fcda4ff5899b6c547dcabff9 P f672de6f5f7e40e9be2569882b1619d7: Deleting WAL directory at /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790376259650642-11032-0/minicluster-data/ts-2/wal/wals/1ffd07f6fcda4ff5899b6c547dcabff9
I20260925 22:44:45.193391 15627 ts_tablet_manager.cc:1918] T 1ffd07f6fcda4ff5899b6c547dcabff9 P b2dfcaefbadc4192ac7b889a1e501d14: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 22:44:45.193516 15627 log.cc:1196] T 1ffd07f6fcda4ff5899b6c547dcabff9 P b2dfcaefbadc4192ac7b889a1e501d14: Deleting WAL directory at /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790376259650642-11032-0/minicluster-data/ts-1/wal/wals/1ffd07f6fcda4ff5899b6c547dcabff9
I20260925 22:44:45.193576 15629 ts_tablet_manager.cc:1939] T 1ffd07f6fcda4ff5899b6c547dcabff9 P f672de6f5f7e40e9be2569882b1619d7: Deleting consensus metadata
I20260925 22:44:45.193823 15627 ts_tablet_manager.cc:1939] T 1ffd07f6fcda4ff5899b6c547dcabff9 P b2dfcaefbadc4192ac7b889a1e501d14: Deleting consensus metadata
I20260925 22:44:45.195046 15116 catalog_manager.cc:4841] TS b2dfcaefbadc4192ac7b889a1e501d14 (127.10.198.2:41879): tablet 1ffd07f6fcda4ff5899b6c547dcabff9 (table test-table [id=9b0f6e7ce5644bb8bab82c895fc3129c]) successfully deleted
I20260925 22:44:45.195322 15116 catalog_manager.cc:4841] TS f672de6f5f7e40e9be2569882b1619d7 (127.10.198.3:46589): tablet 1ffd07f6fcda4ff5899b6c547dcabff9 (table test-table [id=9b0f6e7ce5644bb8bab82c895fc3129c]) successfully deleted
W20260925 22:44:45.196405 15115 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.10.198.1:43219 (error 108)
I20260925 22:44:45.197221 11032 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrWXm8X/build/debug/bin/kudu with pid 15325
I20260925 22:44:45.197608 15598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:44:45.197231 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.198.1:43219 (local address 127.0.0.1:34094)
0925 22:44:45.197406 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0925 22:44:45.197494 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.198.1:43219: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
W20260925 22:44:45.198096 15115 catalog_manager.cc:4584] TS 117d64e10a9b4976a84373ccf4851c0e (127.10.198.1:43219): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 1ffd07f6fcda4ff5899b6c547dcabff9: Network error: Client connection negotiation failed: client connection to 127.10.198.1:43219: connect: Connection refused (error 111)
I20260925 22:44:45.203022 11032 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrWXm8X/build/debug/bin/kudu with pid 15458
I20260925 22:44:45.208849 11032 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrWXm8X/build/debug/bin/kudu with pid 15099
2026-09-25T22:44:45Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (2521 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-09-25T22:44:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T22:44:45Z Disabled control of system clock
I20260925 22:44:45.241338 11032 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrWXm8X/build/debug/bin/kudu
/tmp/dist-test-taskrWXm8X/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790376259650642-11032-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790376259650642-11032-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790376259650642-11032-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790376259650642-11032-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.198.62:43407
--webserver_interface=127.10.198.62
--webserver_port=0
--builtin_ntp_servers=127.10.198.20:33251
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.198.62:43407
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260925 22:44:45.362221 15634 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:44:45.362550 15634 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:44:45.362591 15634 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:44:45.367003 15634 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 22:44:45.367101 15634 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:44:45.367117 15634 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:44:45.367136 15634 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 22:44:45.367151 15634 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 22:44:45.372581 15634 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.198.20:33251
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790376259650642-11032-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790376259650642-11032-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.198.62:43407
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.198.62:43407
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790376259650642-11032-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.198.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15634
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790376259650642-11032-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:09:16 UTC on ccycloud.quasar-afmhyw.root.comops.site
build id 4673379
I20260925 22:44:45.373835 15634 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:44:45.375372 15634 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:44:45.384675 15640 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:44:45.384670 15642 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:44:45.384670 15639 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:44:45.385038 15634 server_base.cc:1034] running on GCE node
I20260925 22:44:45.385826 15634 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:44:45.387117 15634 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:44:45.388330 15634 hybrid_clock.cc:648] HybridClock initialized: now 1790376285388292 us; error 59 us; skew 500 ppm
I20260925 22:44:45.390637 15634 webserver.cc:460] Webserver started at http://127.10.198.62:42743/ using document root <none> and password file <none>
I20260925 22:44:45.391654 15634 fs_manager.cc:359] Metadata directory not provided
I20260925 22:44:45.391788 15634 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:44:45.392136 15634 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:44:45.394479 15634 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790376259650642-11032-0/minicluster-data/master-0/data/instance:
uuid: "e3469d866f62462fb807f2085654bd93"
format_stamp: "Formatted at 2026-09-25 22:44:45 on dist-test-slave-5vdr"
server_key: "cd373e0d3120593e7ce6a405cd0ba52a"
server_key_iv: "17c1ef70911bf9fd43826728e161220a"
server_key_version: "encryptionkey@0"
I20260925 22:44:45.395156 15634 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790376259650642-11032-0/minicluster-data/master-0/wal/instance:
uuid: "e3469d866f62462fb807f2085654bd93"
format_stamp: "Formatted at 2026-09-25 22:44:45 on dist-test-slave-5vdr"
server_key: "cd373e0d3120593e7ce6a405cd0ba52a"
server_key_iv: "17c1ef70911bf9fd43826728e161220a"
server_key_version: "encryptionkey@0"
I20260925 22:44:45.400063 15634 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260925 22:44:45.403075 15648 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:44:45.404654 15634 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 22:44:45.404842 15634 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790376259650642-11032-0/minicluster-data/master-0/data,/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790376259650642-11032-0/minicluster-data/master-0/wal
uuid: "e3469d866f62462fb807f2085654bd93"
format_stamp: "Formatted at 2026-09-25 22:44:45 on dist-test-slave-5vdr"
server_key: "cd373e0d3120593e7ce6a405cd0ba52a"
server_key_iv: "17c1ef70911bf9fd43826728e161220a"
server_key_version: "encryptionkey@0"
I20260925 22:44:45.404996 15634 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790376259650642-11032-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790376259650642-11032-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790376259650642-11032-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:44:45.410248 15634 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:44:45.411010 15634 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:44:45.411233 15634 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:44:45.419216 15634 rpc_server.cc:307] RPC server started. Bound to: 127.10.198.62:43407
I20260925 22:44:45.419221 15700 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.198.62:43407 every 8 connection(s)
I20260925 22:44:45.420384 15634 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790376259650642-11032-0/minicluster-data/master-0/data/info.pb
I20260925 22:44:45.423945 15701 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:44:45.428831 11032 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrWXm8X/build/debug/bin/kudu as pid 15634
I20260925 22:44:45.429001 11032 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790376259650642-11032-0/minicluster-data/master-0/wal/instance
I20260925 22:44:45.429318 11032 external_mini_cluster.cc:1428] Setting key e71d14271b0a731456cc8e2fe7218f00
I20260925 22:44:45.430580 15701 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e3469d866f62462fb807f2085654bd93: Bootstrap starting.
I20260925 22:44:45.433348 15701 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e3469d866f62462fb807f2085654bd93: Neither blocks nor log segments found. Creating new log.
I20260925 22:44:45.434389 15701 log.cc:824] T 00000000000000000000000000000000 P e3469d866f62462fb807f2085654bd93: Log is configured to *not* fsync() on all Append() calls
I20260925 22:44:45.435716 15704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:44:45.431007 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41066 (local address 127.10.198.62:43407)
0925 22:44:45.431720 (+ 713us) server_negotiation.cc:206] Beginning negotiation
0925 22:44:45.431737 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0925 22:44:45.431778 (+ 41us) server_negotiation.cc:407] Connection header received
0925 22:44:45.432583 (+ 805us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:44:45.432602 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:44:45.433016 (+ 414us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:44:45.433349 (+ 333us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:44:45.434265 (+ 916us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:44:45.434271 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:44:45.434285 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 22:44:45.434296 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:44:45.434352 (+ 56us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 22:44:45.434368 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:44:45.434464 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0925 22:44:45.434639 (+ 175us) server_negotiation.cc:299] Negotiation successful
0925 22:44:45.434960 (+ 321us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":308,"thread_start_us":177,"threads_started":1}
I20260925 22:44:45.436738 15701 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e3469d866f62462fb807f2085654bd93: No bootstrap required, opened a new log
I20260925 22:44:45.440923 15701 raft_consensus.cc:348] T 00000000000000000000000000000000 P e3469d866f62462fb807f2085654bd93 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e3469d866f62462fb807f2085654bd93" member_type: VOTER last_known_addr { host: "127.10.198.62" port: 43407 } }
I20260925 22:44:45.441242 15701 raft_consensus.cc:374] T 00000000000000000000000000000000 P e3469d866f62462fb807f2085654bd93 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:44:45.441313 15701 raft_consensus.cc:729] T 00000000000000000000000000000000 P e3469d866f62462fb807f2085654bd93 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e3469d866f62462fb807f2085654bd93, State: Initialized, Role: FOLLOWER
I20260925 22:44:45.441833 15701 consensus_queue.cc:260] T 00000000000000000000000000000000 P e3469d866f62462fb807f2085654bd93 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e3469d866f62462fb807f2085654bd93" member_type: VOTER last_known_addr { host: "127.10.198.62" port: 43407 } }
I20260925 22:44:45.441995 15701 raft_consensus.cc:388] T 00000000000000000000000000000000 P e3469d866f62462fb807f2085654bd93 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 22:44:45.442068 15701 raft_consensus.cc:482] T 00000000000000000000000000000000 P e3469d866f62462fb807f2085654bd93 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 22:44:45.442203 15701 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e3469d866f62462fb807f2085654bd93 [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:44:45.443287 15701 raft_consensus.cc:504] T 00000000000000000000000000000000 P e3469d866f62462fb807f2085654bd93 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e3469d866f62462fb807f2085654bd93" member_type: VOTER last_known_addr { host: "127.10.198.62" port: 43407 } }
I20260925 22:44:45.443702 15701 leader_election.cc:302] T 00000000000000000000000000000000 P e3469d866f62462fb807f2085654bd93 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e3469d866f62462fb807f2085654bd93; no voters:
I20260925 22:44:45.444114 15701 leader_election.cc:288] T 00000000000000000000000000000000 P e3469d866f62462fb807f2085654bd93 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 22:44:45.444262 15706 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e3469d866f62462fb807f2085654bd93 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 22:44:45.444551 15706 raft_consensus.cc:686] T 00000000000000000000000000000000 P e3469d866f62462fb807f2085654bd93 [term 1 LEADER]: Becoming Leader. State: Replica: e3469d866f62462fb807f2085654bd93, State: Running, Role: LEADER
I20260925 22:44:45.444958 15706 consensus_queue.cc:237] T 00000000000000000000000000000000 P e3469d866f62462fb807f2085654bd93 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e3469d866f62462fb807f2085654bd93" member_type: VOTER last_known_addr { host: "127.10.198.62" port: 43407 } }
I20260925 22:44:45.445601 15701 sys_catalog.cc:564] T 00000000000000000000000000000000 P e3469d866f62462fb807f2085654bd93 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 22:44:45.446949 15707 sys_catalog.cc:455] T 00000000000000000000000000000000 P e3469d866f62462fb807f2085654bd93 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e3469d866f62462fb807f2085654bd93" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e3469d866f62462fb807f2085654bd93" member_type: VOTER last_known_addr { host: "127.10.198.62" port: 43407 } } }
I20260925 22:44:45.447132 15707 sys_catalog.cc:458] T 00000000000000000000000000000000 P e3469d866f62462fb807f2085654bd93 [sys.catalog]: This master's current role is: LEADER
I20260925 22:44:45.446964 15708 sys_catalog.cc:455] T 00000000000000000000000000000000 P e3469d866f62462fb807f2085654bd93 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e3469d866f62462fb807f2085654bd93. Latest consensus state: current_term: 1 leader_uuid: "e3469d866f62462fb807f2085654bd93" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e3469d866f62462fb807f2085654bd93" member_type: VOTER last_known_addr { host: "127.10.198.62" port: 43407 } } }
I20260925 22:44:45.447271 15708 sys_catalog.cc:458] T 00000000000000000000000000000000 P e3469d866f62462fb807f2085654bd93 [sys.catalog]: This master's current role is: LEADER
I20260925 22:44:45.447613 15715 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 22:44:45.451156 15715 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 22:44:45.457592 15715 catalog_manager.cc:1347] Generated new cluster ID: ee625899f9954bd5b9f8173033c593c7
I20260925 22:44:45.457705 15715 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 22:44:45.465210 15715 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 22:44:45.465631 15715 catalog_manager.cc:1504] Loading token signing keys...
I20260925 22:44:45.472442 15715 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e3469d866f62462fb807f2085654bd93: Generated new TSK 0
I20260925 22:44:45.473255 15715 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 22:44:45.479209 11032 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrWXm8X/build/debug/bin/kudu
/tmp/dist-test-taskrWXm8X/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790376259650642-11032-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790376259650642-11032-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790376259650642-11032-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790376259650642-11032-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.198.1:0
--local_ip_for_outbound_sockets=127.10.198.1
--webserver_interface=127.10.198.1
--webserver_port=0
--tserver_master_addrs=127.10.198.62:43407
--builtin_ntp_servers=127.10.198.20:33251
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260925 22:44:45.605962 15725 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:44:45.606284 15725 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:44:45.606329 15725 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:44:45.610932 15725 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:44:45.611027 15725 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:44:45.611089 15725 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.198.1
I20260925 22:44:45.616416 15725 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.198.20:33251
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790376259650642-11032-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790376259650642-11032-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.198.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790376259650642-11032-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.198.1
--webserver_port=0
--tserver_master_addrs=127.10.198.62:43407
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15725
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.198.1
--log_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790376259650642-11032-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:09:16 UTC on ccycloud.quasar-afmhyw.root.comops.site
build id 4673379
I20260925 22:44:45.617614 15725 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:44:45.619179 15725 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:44:45.629942 15733 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:44:45.630039 15730 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:44:45.629942 15731 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:44:45.630203 15725 server_base.cc:1034] running on GCE node
I20260925 22:44:45.630929 15725 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:44:45.631790 15725 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:44:45.633068 15725 hybrid_clock.cc:648] HybridClock initialized: now 1790376285633047 us; error 46 us; skew 500 ppm
I20260925 22:44:45.635412 15725 webserver.cc:460] Webserver started at http://127.10.198.1:40815/ using document root <none> and password file <none>
I20260925 22:44:45.636234 15725 fs_manager.cc:359] Metadata directory not provided
I20260925 22:44:45.636317 15725 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:44:45.636561 15725 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:44:45.638908 15725 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790376259650642-11032-0/minicluster-data/ts-0/data/instance:
uuid: "11c3cbe22512415a955dc1b70010e3f0"
format_stamp: "Formatted at 2026-09-25 22:44:45 on dist-test-slave-5vdr"
server_key: "75d967602ddcb8bf42dc5f98ec9af825"
server_key_iv: "955d44e49585aadb3c5e52f2bbd65c40"
server_key_version: "encryptionkey@0"
I20260925 22:44:45.639627 15725 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790376259650642-11032-0/minicluster-data/ts-0/wal/instance:
uuid: "11c3cbe22512415a955dc1b70010e3f0"
format_stamp: "Formatted at 2026-09-25 22:44:45 on dist-test-slave-5vdr"
server_key: "75d967602ddcb8bf42dc5f98ec9af825"
server_key_iv: "955d44e49585aadb3c5e52f2bbd65c40"
server_key_version: "encryptionkey@0"
I20260925 22:44:45.644187 15725 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.004s
I20260925 22:44:45.647575 15739 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:44:45.649245 15725 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260925 22:44:45.649427 15725 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790376259650642-11032-0/minicluster-data/ts-0/data,/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790376259650642-11032-0/minicluster-data/ts-0/wal
uuid: "11c3cbe22512415a955dc1b70010e3f0"
format_stamp: "Formatted at 2026-09-25 22:44:45 on dist-test-slave-5vdr"
server_key: "75d967602ddcb8bf42dc5f98ec9af825"
server_key_iv: "955d44e49585aadb3c5e52f2bbd65c40"
server_key_version: "encryptionkey@0"
I20260925 22:44:45.649587 15725 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790376259650642-11032-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790376259650642-11032-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790376259650642-11032-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:44:45.654855 15725 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:44:45.655608 15725 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:44:45.655910 15725 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:44:45.656621 15725 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 22:44:45.657702 15725 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:44:45.657755 15725 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:44:45.657799 15725 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:44:45.657857 15725 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 22:44:45.667637 15725 rpc_server.cc:307] RPC server started. Bound to: 127.10.198.1:33801
I20260925 22:44:45.667665 15852 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.198.1:33801 every 8 connection(s)
I20260925 22:44:45.668785 15725 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790376259650642-11032-0/minicluster-data/ts-0/data/info.pb
I20260925 22:44:45.675802 15704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:44:45.670984 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.198.1:38585 (local address 127.10.198.62:43407)
0925 22:44:45.671177 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0925 22:44:45.671182 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:44:45.672524 (+ 1342us) server_negotiation.cc:407] Connection header received
0925 22:44:45.673941 (+ 1417us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:44:45.673945 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:44:45.673989 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:44:45.674073 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:44:45.675318 (+ 1245us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:44:45.675322 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:44:45.675324 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 22:44:45.675339 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:44:45.675358 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 22:44:45.675367 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:44:45.675465 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0925 22:44:45.675624 (+ 159us) server_negotiation.cc:299] Negotiation successful
0925 22:44:45.675672 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260925 22:44:45.676695 11032 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrWXm8X/build/debug/bin/kudu as pid 15725
I20260925 22:44:45.676864 11032 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790376259650642-11032-0/minicluster-data/ts-0/wal/instance
I20260925 22:44:45.676944 15855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:44:45.671428 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.198.62:43407 (local address 127.10.198.1:38585)
0925 22:44:45.672335 (+ 907us) negotiation.cc:107] Waiting for socket to connect
0925 22:44:45.672387 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0925 22:44:45.673681 (+ 1294us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:44:45.674282 (+ 601us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:44:45.674291 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:44:45.674394 (+ 103us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:44:45.674999 (+ 605us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 22:44:45.675001 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:44:45.675046 (+ 45us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 22:44:45.675123 (+ 77us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:44:45.675456 (+ 333us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:44:45.675463 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:44:45.675468 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 22:44:45.675619 (+ 151us) client_negotiation.cc:240] Negotiation successful
0925 22:44:45.675931 (+ 312us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":378,"thread_start_us":143,"threads_started":1}
I20260925 22:44:45.677244 11032 external_mini_cluster.cc:1428] Setting key 5ff34d4a07f6929568f675b2c6b0d20f
I20260925 22:44:45.678579 15853 heartbeater.cc:347] Connected to a master server at 127.10.198.62:43407
I20260925 22:44:45.678869 15853 heartbeater.cc:464] Registering TS with master...
I20260925 22:44:45.679419 15853 heartbeater.cc:511] Master 127.10.198.62:43407 requested a full tablet report, sending...
I20260925 22:44:45.679670 11032 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrWXm8X/build/debug/bin/kudu
/tmp/dist-test-taskrWXm8X/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790376259650642-11032-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790376259650642-11032-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790376259650642-11032-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790376259650642-11032-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.198.2:0
--local_ip_for_outbound_sockets=127.10.198.2
--webserver_interface=127.10.198.2
--webserver_port=0
--tserver_master_addrs=127.10.198.62:43407
--builtin_ntp_servers=127.10.198.20:33251
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260925 22:44:45.681354 15664 ts_manager.cc:194] Registered new tserver with Master: 11c3cbe22512415a955dc1b70010e3f0 (127.10.198.1:33801)
W20260925 22:44:45.802356 15856 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:44:45.802688 15856 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:44:45.802732 15856 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:44:45.807531 15856 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:44:45.807632 15856 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:44:45.807699 15856 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.198.2
I20260925 22:44:45.813598 15856 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.198.20:33251
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790376259650642-11032-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790376259650642-11032-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.198.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790376259650642-11032-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.198.2
--webserver_port=0
--tserver_master_addrs=127.10.198.62:43407
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15856
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.198.2
--log_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790376259650642-11032-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:09:16 UTC on ccycloud.quasar-afmhyw.root.comops.site
build id 4673379
I20260925 22:44:45.814849 15856 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:44:45.816438 15856 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:44:45.827450 15862 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:44:45.827450 15864 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:44:45.827466 15861 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:44:45.828222 15856 server_base.cc:1034] running on GCE node
I20260925 22:44:45.828773 15856 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:44:45.829558 15856 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:44:45.830755 15856 hybrid_clock.cc:648] HybridClock initialized: now 1790376285830727 us; error 48 us; skew 500 ppm
I20260925 22:44:45.833140 15856 webserver.cc:460] Webserver started at http://127.10.198.2:39495/ using document root <none> and password file <none>
I20260925 22:44:45.833854 15856 fs_manager.cc:359] Metadata directory not provided
I20260925 22:44:45.833933 15856 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:44:45.834178 15856 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:44:45.836649 15856 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790376259650642-11032-0/minicluster-data/ts-1/data/instance:
uuid: "1c1f35e4e0ab446e8ea73d158b74e17d"
format_stamp: "Formatted at 2026-09-25 22:44:45 on dist-test-slave-5vdr"
server_key: "d9005cf79f0232712a4126b506f9542d"
server_key_iv: "f619cf14feee19f5d168d489ee04b4fb"
server_key_version: "encryptionkey@0"
I20260925 22:44:45.837296 15856 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790376259650642-11032-0/minicluster-data/ts-1/wal/instance:
uuid: "1c1f35e4e0ab446e8ea73d158b74e17d"
format_stamp: "Formatted at 2026-09-25 22:44:45 on dist-test-slave-5vdr"
server_key: "d9005cf79f0232712a4126b506f9542d"
server_key_iv: "f619cf14feee19f5d168d489ee04b4fb"
server_key_version: "encryptionkey@0"
I20260925 22:44:45.842106 15856 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260925 22:44:45.845544 15870 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:44:45.847474 15856 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260925 22:44:45.847641 15856 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790376259650642-11032-0/minicluster-data/ts-1/data,/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790376259650642-11032-0/minicluster-data/ts-1/wal
uuid: "1c1f35e4e0ab446e8ea73d158b74e17d"
format_stamp: "Formatted at 2026-09-25 22:44:45 on dist-test-slave-5vdr"
server_key: "d9005cf79f0232712a4126b506f9542d"
server_key_iv: "f619cf14feee19f5d168d489ee04b4fb"
server_key_version: "encryptionkey@0"
I20260925 22:44:45.847898 15856 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790376259650642-11032-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790376259650642-11032-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790376259650642-11032-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:44:45.853974 15856 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:44:45.854795 15856 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:44:45.854969 15856 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:44:45.855613 15856 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 22:44:45.856820 15856 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:44:45.856874 15856 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:44:45.856959 15856 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:44:45.856988 15856 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 22:44:45.867377 15856 rpc_server.cc:307] RPC server started. Bound to: 127.10.198.2:34757
I20260925 22:44:45.867409 15983 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.198.2:34757 every 8 connection(s)
I20260925 22:44:45.868568 15856 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790376259650642-11032-0/minicluster-data/ts-1/data/info.pb
I20260925 22:44:45.876087 15704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:44:45.870938 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.198.2:60415 (local address 127.10.198.62:43407)
0925 22:44:45.871136 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0925 22:44:45.871143 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 22:44:45.872560 (+ 1417us) server_negotiation.cc:407] Connection header received
0925 22:44:45.873974 (+ 1414us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:44:45.873977 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:44:45.874027 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:44:45.874125 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:44:45.875499 (+ 1374us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:44:45.875503 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:44:45.875505 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 22:44:45.875519 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:44:45.875540 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 22:44:45.875549 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:44:45.875656 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0925 22:44:45.875834 (+ 178us) server_negotiation.cc:299] Negotiation successful
0925 22:44:45.875963 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260925 22:44:45.877264 11032 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrWXm8X/build/debug/bin/kudu as pid 15856
I20260925 22:44:45.877370 11032 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790376259650642-11032-0/minicluster-data/ts-1/wal/instance
I20260925 22:44:45.877413 15986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:44:45.871399 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.198.62:43407 (local address 127.10.198.2:60415)
0925 22:44:45.872372 (+ 973us) negotiation.cc:107] Waiting for socket to connect
0925 22:44:45.872428 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0925 22:44:45.873695 (+ 1267us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:44:45.874328 (+ 633us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:44:45.874336 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:44:45.874442 (+ 106us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:44:45.875156 (+ 714us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 22:44:45.875158 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:44:45.875211 (+ 53us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 22:44:45.875295 (+ 84us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:44:45.875651 (+ 356us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:44:45.875657 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:44:45.875663 (+ 6us) client_negotiation.cc:769] Sending connection context
0925 22:44:45.875895 (+ 232us) client_negotiation.cc:240] Negotiation successful
0925 22:44:45.876141 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":340,"thread_start_us":153,"threads_started":1}
I20260925 22:44:45.877781 11032 external_mini_cluster.cc:1428] Setting key f32a76ddb528185b006b0c9f2cd37e07
I20260925 22:44:45.878809 15984 heartbeater.cc:347] Connected to a master server at 127.10.198.62:43407
I20260925 22:44:45.879118 15984 heartbeater.cc:464] Registering TS with master...
I20260925 22:44:45.879738 15984 heartbeater.cc:511] Master 127.10.198.62:43407 requested a full tablet report, sending...
I20260925 22:44:45.879976 11032 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrWXm8X/build/debug/bin/kudu
/tmp/dist-test-taskrWXm8X/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790376259650642-11032-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790376259650642-11032-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790376259650642-11032-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790376259650642-11032-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.198.3:0
--local_ip_for_outbound_sockets=127.10.198.3
--webserver_interface=127.10.198.3
--webserver_port=0
--tserver_master_addrs=127.10.198.62:43407
--builtin_ntp_servers=127.10.198.20:33251
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260925 22:44:45.881160 15664 ts_manager.cc:194] Registered new tserver with Master: 1c1f35e4e0ab446e8ea73d158b74e17d (127.10.198.2:34757)
W20260925 22:44:46.002774 15987 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:44:46.003095 15987 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:44:46.003139 15987 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:44:46.007511 15987 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:44:46.007598 15987 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:44:46.007663 15987 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.198.3
I20260925 22:44:46.013357 15987 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.198.20:33251
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790376259650642-11032-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790376259650642-11032-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.198.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790376259650642-11032-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.198.3
--webserver_port=0
--tserver_master_addrs=127.10.198.62:43407
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15987
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.198.3
--log_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790376259650642-11032-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:09:16 UTC on ccycloud.quasar-afmhyw.root.comops.site
build id 4673379
I20260925 22:44:46.014622 15987 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:44:46.016266 15987 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:44:46.028311 15993 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:44:46.028311 15992 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:44:46.028231 15995 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:44:46.028651 15987 server_base.cc:1034] running on GCE node
I20260925 22:44:46.029227 15987 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:44:46.029927 15987 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:44:46.031159 15987 hybrid_clock.cc:648] HybridClock initialized: now 1790376286031137 us; error 43 us; skew 500 ppm
I20260925 22:44:46.033535 15987 webserver.cc:460] Webserver started at http://127.10.198.3:34819/ using document root <none> and password file <none>
I20260925 22:44:46.034232 15987 fs_manager.cc:359] Metadata directory not provided
I20260925 22:44:46.034350 15987 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:44:46.034608 15987 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:44:46.037093 15987 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790376259650642-11032-0/minicluster-data/ts-2/data/instance:
uuid: "307e12871adf4718a24b3a48edad7b09"
format_stamp: "Formatted at 2026-09-25 22:44:46 on dist-test-slave-5vdr"
server_key: "387f5ed399117dff84c26952d6857291"
server_key_iv: "db253837557c625fd34ee563b3fecaf9"
server_key_version: "encryptionkey@0"
I20260925 22:44:46.037762 15987 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790376259650642-11032-0/minicluster-data/ts-2/wal/instance:
uuid: "307e12871adf4718a24b3a48edad7b09"
format_stamp: "Formatted at 2026-09-25 22:44:46 on dist-test-slave-5vdr"
server_key: "387f5ed399117dff84c26952d6857291"
server_key_iv: "db253837557c625fd34ee563b3fecaf9"
server_key_version: "encryptionkey@0"
I20260925 22:44:46.042135 15987 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260925 22:44:46.045212 16001 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:44:46.046926 15987 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260925 22:44:46.047091 15987 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790376259650642-11032-0/minicluster-data/ts-2/data,/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790376259650642-11032-0/minicluster-data/ts-2/wal
uuid: "307e12871adf4718a24b3a48edad7b09"
format_stamp: "Formatted at 2026-09-25 22:44:46 on dist-test-slave-5vdr"
server_key: "387f5ed399117dff84c26952d6857291"
server_key_iv: "db253837557c625fd34ee563b3fecaf9"
server_key_version: "encryptionkey@0"
I20260925 22:44:46.047255 15987 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790376259650642-11032-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790376259650642-11032-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790376259650642-11032-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:44:46.052583 15987 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:44:46.053390 15987 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:44:46.053624 15987 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:44:46.054296 15987 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 22:44:46.055433 15987 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:44:46.055510 15987 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:44:46.055585 15987 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:44:46.055629 15987 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 22:44:46.065279 15987 rpc_server.cc:307] RPC server started. Bound to: 127.10.198.3:42461
I20260925 22:44:46.065305 16114 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.198.3:42461 every 8 connection(s)
I20260925 22:44:46.066334 15987 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790376259650642-11032-0/minicluster-data/ts-2/data/info.pb
I20260925 22:44:46.067134 11032 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrWXm8X/build/debug/bin/kudu as pid 15987
I20260925 22:44:46.067269 11032 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790376259650642-11032-0/minicluster-data/ts-2/wal/instance
I20260925 22:44:46.067553 11032 external_mini_cluster.cc:1428] Setting key 125574f9b33b57d5aee84378fcaf58bb
I20260925 22:44:46.073807 15704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:44:46.070249 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.198.3:37151 (local address 127.10.198.62:43407)
0925 22:44:46.070403 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0925 22:44:46.070408 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:44:46.070423 (+ 15us) server_negotiation.cc:407] Connection header received
0925 22:44:46.071752 (+ 1329us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:44:46.071756 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:44:46.071804 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:44:46.071917 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:44:46.073266 (+ 1349us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:44:46.073270 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:44:46.073275 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 22:44:46.073295 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:44:46.073317 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 22:44:46.073333 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:44:46.073446 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0925 22:44:46.073596 (+ 150us) server_negotiation.cc:299] Negotiation successful
0925 22:44:46.073648 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260925 22:44:46.074944 16117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:44:46.069203 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.198.62:43407 (local address 127.10.198.3:37151)
0925 22:44:46.070148 (+ 945us) negotiation.cc:107] Waiting for socket to connect
0925 22:44:46.070201 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0925 22:44:46.071497 (+ 1296us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:44:46.072109 (+ 612us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:44:46.072117 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:44:46.072224 (+ 107us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:44:46.072917 (+ 693us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 22:44:46.072918 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:44:46.072968 (+ 50us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 22:44:46.073063 (+ 95us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:44:46.073425 (+ 362us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:44:46.073434 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:44:46.073440 (+ 6us) client_negotiation.cc:769] Sending connection context
0925 22:44:46.073653 (+ 213us) client_negotiation.cc:240] Negotiation successful
0925 22:44:46.073960 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":350,"thread_start_us":160,"threads_started":1}
I20260925 22:44:46.076933 16115 heartbeater.cc:347] Connected to a master server at 127.10.198.62:43407
I20260925 22:44:46.077406 16115 heartbeater.cc:464] Registering TS with master...
I20260925 22:44:46.077979 16115 heartbeater.cc:511] Master 127.10.198.62:43407 requested a full tablet report, sending...
I20260925 22:44:46.079427 15664 ts_manager.cc:194] Registered new tserver with Master: 307e12871adf4718a24b3a48edad7b09 (127.10.198.3:42461)
I20260925 22:44:46.082460 11032 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 22:44:46.087500 15704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:44:46.085346 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41082 (local address 127.10.198.62:43407)
0925 22:44:46.085516 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0925 22:44:46.085521 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:44:46.085682 (+ 161us) server_negotiation.cc:407] Connection header received
0925 22:44:46.085870 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:44:46.085874 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:44:46.085926 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:44:46.086020 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:44:46.086830 (+ 810us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:44:46.086834 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:44:46.086842 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 22:44:46.086862 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:44:46.086934 (+ 72us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 22:44:46.086967 (+ 33us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:44:46.087159 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0925 22:44:46.087255 (+ 96us) server_negotiation.cc:299] Negotiation successful
0925 22:44:46.087310 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260925 22:44:46.091655 15664 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41082:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 22:44:46.095108 15664 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 22:44:46.109117 16126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:44:46.105489 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.198.1:33801 (local address 127.0.0.1:38020)
0925 22:44:46.106526 (+ 1037us) negotiation.cc:107] Waiting for socket to connect
0925 22:44:46.106543 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 22:44:46.106672 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:44:46.107273 (+ 601us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:44:46.107283 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:44:46.107381 (+ 98us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:44:46.108277 (+ 896us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 22:44:46.108279 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:44:46.108291 (+ 12us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 22:44:46.108349 (+ 58us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:44:46.108748 (+ 399us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:44:46.108751 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:44:46.108753 (+ 2us) client_negotiation.cc:769] Sending connection context
0925 22:44:46.108838 (+ 85us) client_negotiation.cc:240] Negotiation successful
0925 22:44:46.108947 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":990,"mutex_wait_us":23,"spinlock_wait_cycles":2304,"thread_start_us":108,"threads_started":1}
I20260925 22:44:46.109162 16130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:44:46.106269 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37342 (local address 127.10.198.3:42461)
0925 22:44:46.106967 (+ 698us) server_negotiation.cc:206] Beginning negotiation
0925 22:44:46.106978 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 22:44:46.107001 (+ 23us) server_negotiation.cc:407] Connection header received
0925 22:44:46.107052 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:44:46.107057 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:44:46.107233 (+ 176us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:44:46.107706 (+ 473us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:44:46.108463 (+ 757us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:44:46.108473 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:44:46.108488 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 22:44:46.108502 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:44:46.108546 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 22:44:46.108577 (+ 31us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:44:46.108705 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0925 22:44:46.108883 (+ 178us) server_negotiation.cc:299] Negotiation successful
0925 22:44:46.108975 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":579,"thread_start_us":139,"threads_started":1}
I20260925 22:44:46.109151 16128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:44:46.105472 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.198.3:42461 (local address 127.0.0.1:37342)
0925 22:44:46.106481 (+ 1009us) negotiation.cc:107] Waiting for socket to connect
0925 22:44:46.106544 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0925 22:44:46.106672 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:44:46.107372 (+ 700us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:44:46.107375 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:44:46.107385 (+ 10us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:44:46.108157 (+ 772us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 22:44:46.108159 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:44:46.108182 (+ 23us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 22:44:46.108267 (+ 85us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:44:46.108697 (+ 430us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:44:46.108707 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:44:46.108712 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 22:44:46.108838 (+ 126us) client_negotiation.cc:240] Negotiation successful
0925 22:44:46.108933 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":809,"spinlock_wait_cycles":10368,"thread_start_us":95,"threads_started":1}
I20260925 22:44:46.109644 16131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:44:46.106381 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38020 (local address 127.10.198.1:33801)
0925 22:44:46.106781 (+ 400us) server_negotiation.cc:206] Beginning negotiation
0925 22:44:46.106794 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0925 22:44:46.106818 (+ 24us) server_negotiation.cc:407] Connection header received
0925 22:44:46.106873 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:44:46.106879 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:44:46.107109 (+ 230us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:44:46.107281 (+ 172us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:44:46.108492 (+ 1211us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:44:46.108499 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:44:46.108510 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 22:44:46.108522 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:44:46.108565 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 22:44:46.108600 (+ 35us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:44:46.108711 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0925 22:44:46.109387 (+ 676us) server_negotiation.cc:299] Negotiation successful
0925 22:44:46.109467 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":297,"thread_start_us":121,"threads_started":1}
I20260925 22:44:46.109920 16129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:44:46.105949 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39748 (local address 127.10.198.2:34757)
0925 22:44:46.106402 (+ 453us) server_negotiation.cc:206] Beginning negotiation
0925 22:44:46.106415 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0925 22:44:46.107152 (+ 737us) server_negotiation.cc:407] Connection header received
0925 22:44:46.107632 (+ 480us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:44:46.107641 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:44:46.107865 (+ 224us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:44:46.107976 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:44:46.109231 (+ 1255us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:44:46.109239 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:44:46.109254 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 22:44:46.109268 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:44:46.109329 (+ 61us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 22:44:46.109389 (+ 60us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:44:46.109477 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0925 22:44:46.109668 (+ 191us) server_negotiation.cc:299] Negotiation successful
0925 22:44:46.109744 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":334,"thread_start_us":130,"threads_started":1}
I20260925 22:44:46.110319 16127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:44:46.105782 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.198.2:34757 (local address 127.0.0.1:39748)
0925 22:44:46.107047 (+ 1265us) negotiation.cc:107] Waiting for socket to connect
0925 22:44:46.107061 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 22:44:46.107483 (+ 422us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:44:46.108522 (+ 1039us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:44:46.108525 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:44:46.108542 (+ 17us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:44:46.109042 (+ 500us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 22:44:46.109044 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:44:46.109055 (+ 11us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 22:44:46.109080 (+ 25us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:44:46.109551 (+ 471us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:44:46.109554 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:44:46.109556 (+ 2us) client_negotiation.cc:769] Sending connection context
0925 22:44:46.110131 (+ 575us) client_negotiation.cc:240] Negotiation successful
0925 22:44:46.110170 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1204,"thread_start_us":131,"threads_started":1}
I20260925 22:44:46.113066 15917 tablet_service.cc:1467] Processing CreateTablet for tablet 924cdcf581904c258fe67b288c5467d9 (DEFAULT_TABLE table=test-table [id=4f508aef195b49fe945f3c6df7cb2728]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 22:44:46.113150 16049 tablet_service.cc:1467] Processing CreateTablet for tablet 924cdcf581904c258fe67b288c5467d9 (DEFAULT_TABLE table=test-table [id=4f508aef195b49fe945f3c6df7cb2728]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 22:44:46.113152 15787 tablet_service.cc:1467] Processing CreateTablet for tablet 924cdcf581904c258fe67b288c5467d9 (DEFAULT_TABLE table=test-table [id=4f508aef195b49fe945f3c6df7cb2728]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 22:44:46.114396 15917 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 924cdcf581904c258fe67b288c5467d9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:44:46.114398 15787 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 924cdcf581904c258fe67b288c5467d9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:44:46.114398 16049 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 924cdcf581904c258fe67b288c5467d9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:44:46.121788 16132 tablet_bootstrap.cc:492] T 924cdcf581904c258fe67b288c5467d9 P 1c1f35e4e0ab446e8ea73d158b74e17d: Bootstrap starting.
I20260925 22:44:46.123342 16133 tablet_bootstrap.cc:492] T 924cdcf581904c258fe67b288c5467d9 P 11c3cbe22512415a955dc1b70010e3f0: Bootstrap starting.
I20260925 22:44:46.124245 16134 tablet_bootstrap.cc:492] T 924cdcf581904c258fe67b288c5467d9 P 307e12871adf4718a24b3a48edad7b09: Bootstrap starting.
I20260925 22:44:46.125697 16132 tablet_bootstrap.cc:654] T 924cdcf581904c258fe67b288c5467d9 P 1c1f35e4e0ab446e8ea73d158b74e17d: Neither blocks nor log segments found. Creating new log.
I20260925 22:44:46.125713 16133 tablet_bootstrap.cc:654] T 924cdcf581904c258fe67b288c5467d9 P 11c3cbe22512415a955dc1b70010e3f0: Neither blocks nor log segments found. Creating new log.
I20260925 22:44:46.126567 16134 tablet_bootstrap.cc:654] T 924cdcf581904c258fe67b288c5467d9 P 307e12871adf4718a24b3a48edad7b09: Neither blocks nor log segments found. Creating new log.
I20260925 22:44:46.126652 16133 log.cc:824] T 924cdcf581904c258fe67b288c5467d9 P 11c3cbe22512415a955dc1b70010e3f0: Log is configured to *not* fsync() on all Append() calls
I20260925 22:44:46.126683 16132 log.cc:824] T 924cdcf581904c258fe67b288c5467d9 P 1c1f35e4e0ab446e8ea73d158b74e17d: Log is configured to *not* fsync() on all Append() calls
I20260925 22:44:46.127785 16134 log.cc:824] T 924cdcf581904c258fe67b288c5467d9 P 307e12871adf4718a24b3a48edad7b09: Log is configured to *not* fsync() on all Append() calls
I20260925 22:44:46.129195 16132 tablet_bootstrap.cc:492] T 924cdcf581904c258fe67b288c5467d9 P 1c1f35e4e0ab446e8ea73d158b74e17d: No bootstrap required, opened a new log
I20260925 22:44:46.129205 16133 tablet_bootstrap.cc:492] T 924cdcf581904c258fe67b288c5467d9 P 11c3cbe22512415a955dc1b70010e3f0: No bootstrap required, opened a new log
I20260925 22:44:46.129459 16132 ts_tablet_manager.cc:1397] T 924cdcf581904c258fe67b288c5467d9 P 1c1f35e4e0ab446e8ea73d158b74e17d: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.000s
I20260925 22:44:46.129462 16133 ts_tablet_manager.cc:1397] T 924cdcf581904c258fe67b288c5467d9 P 11c3cbe22512415a955dc1b70010e3f0: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260925 22:44:46.130074 16134 tablet_bootstrap.cc:492] T 924cdcf581904c258fe67b288c5467d9 P 307e12871adf4718a24b3a48edad7b09: No bootstrap required, opened a new log
I20260925 22:44:46.130594 16134 ts_tablet_manager.cc:1397] T 924cdcf581904c258fe67b288c5467d9 P 307e12871adf4718a24b3a48edad7b09: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260925 22:44:46.133554 16132 raft_consensus.cc:348] T 924cdcf581904c258fe67b288c5467d9 P 1c1f35e4e0ab446e8ea73d158b74e17d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "307e12871adf4718a24b3a48edad7b09" member_type: VOTER last_known_addr { host: "127.10.198.3" port: 42461 } } peers { permanent_uuid: "11c3cbe22512415a955dc1b70010e3f0" member_type: VOTER last_known_addr { host: "127.10.198.1" port: 33801 } } peers { permanent_uuid: "1c1f35e4e0ab446e8ea73d158b74e17d" member_type: VOTER last_known_addr { host: "127.10.198.2" port: 34757 } }
I20260925 22:44:46.133878 16132 raft_consensus.cc:374] T 924cdcf581904c258fe67b288c5467d9 P 1c1f35e4e0ab446e8ea73d158b74e17d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:44:46.133935 16132 raft_consensus.cc:729] T 924cdcf581904c258fe67b288c5467d9 P 1c1f35e4e0ab446e8ea73d158b74e17d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c1f35e4e0ab446e8ea73d158b74e17d, State: Initialized, Role: FOLLOWER
I20260925 22:44:46.134447 16132 consensus_queue.cc:260] T 924cdcf581904c258fe67b288c5467d9 P 1c1f35e4e0ab446e8ea73d158b74e17d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "307e12871adf4718a24b3a48edad7b09" member_type: VOTER last_known_addr { host: "127.10.198.3" port: 42461 } } peers { permanent_uuid: "11c3cbe22512415a955dc1b70010e3f0" member_type: VOTER last_known_addr { host: "127.10.198.1" port: 33801 } } peers { permanent_uuid: "1c1f35e4e0ab446e8ea73d158b74e17d" member_type: VOTER last_known_addr { host: "127.10.198.2" port: 34757 } }
I20260925 22:44:46.135190 15984 heartbeater.cc:503] Master 127.10.198.62:43407 was elected leader, sending a full tablet report...
I20260925 22:44:46.135071 16133 raft_consensus.cc:348] T 924cdcf581904c258fe67b288c5467d9 P 11c3cbe22512415a955dc1b70010e3f0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "307e12871adf4718a24b3a48edad7b09" member_type: VOTER last_known_addr { host: "127.10.198.3" port: 42461 } } peers { permanent_uuid: "11c3cbe22512415a955dc1b70010e3f0" member_type: VOTER last_known_addr { host: "127.10.198.1" port: 33801 } } peers { permanent_uuid: "1c1f35e4e0ab446e8ea73d158b74e17d" member_type: VOTER last_known_addr { host: "127.10.198.2" port: 34757 } }
I20260925 22:44:46.135411 16133 raft_consensus.cc:374] T 924cdcf581904c258fe67b288c5467d9 P 11c3cbe22512415a955dc1b70010e3f0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:44:46.135478 16133 raft_consensus.cc:729] T 924cdcf581904c258fe67b288c5467d9 P 11c3cbe22512415a955dc1b70010e3f0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 11c3cbe22512415a955dc1b70010e3f0, State: Initialized, Role: FOLLOWER
I20260925 22:44:46.136006 16133 consensus_queue.cc:260] T 924cdcf581904c258fe67b288c5467d9 P 11c3cbe22512415a955dc1b70010e3f0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "307e12871adf4718a24b3a48edad7b09" member_type: VOTER last_known_addr { host: "127.10.198.3" port: 42461 } } peers { permanent_uuid: "11c3cbe22512415a955dc1b70010e3f0" member_type: VOTER last_known_addr { host: "127.10.198.1" port: 33801 } } peers { permanent_uuid: "1c1f35e4e0ab446e8ea73d158b74e17d" member_type: VOTER last_known_addr { host: "127.10.198.2" port: 34757 } }
I20260925 22:44:46.136018 16134 raft_consensus.cc:348] T 924cdcf581904c258fe67b288c5467d9 P 307e12871adf4718a24b3a48edad7b09 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "307e12871adf4718a24b3a48edad7b09" member_type: VOTER last_known_addr { host: "127.10.198.3" port: 42461 } } peers { permanent_uuid: "11c3cbe22512415a955dc1b70010e3f0" member_type: VOTER last_known_addr { host: "127.10.198.1" port: 33801 } } peers { permanent_uuid: "1c1f35e4e0ab446e8ea73d158b74e17d" member_type: VOTER last_known_addr { host: "127.10.198.2" port: 34757 } }
I20260925 22:44:46.136298 16134 raft_consensus.cc:374] T 924cdcf581904c258fe67b288c5467d9 P 307e12871adf4718a24b3a48edad7b09 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:44:46.136372 16134 raft_consensus.cc:729] T 924cdcf581904c258fe67b288c5467d9 P 307e12871adf4718a24b3a48edad7b09 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 307e12871adf4718a24b3a48edad7b09, State: Initialized, Role: FOLLOWER
I20260925 22:44:46.136823 16134 consensus_queue.cc:260] T 924cdcf581904c258fe67b288c5467d9 P 307e12871adf4718a24b3a48edad7b09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "307e12871adf4718a24b3a48edad7b09" member_type: VOTER last_known_addr { host: "127.10.198.3" port: 42461 } } peers { permanent_uuid: "11c3cbe22512415a955dc1b70010e3f0" member_type: VOTER last_known_addr { host: "127.10.198.1" port: 33801 } } peers { permanent_uuid: "1c1f35e4e0ab446e8ea73d158b74e17d" member_type: VOTER last_known_addr { host: "127.10.198.2" port: 34757 } }
I20260925 22:44:46.137279 16133 ts_tablet_manager.cc:1428] T 924cdcf581904c258fe67b288c5467d9 P 11c3cbe22512415a955dc1b70010e3f0: Time spent starting tablet: real 0.008s user 0.009s sys 0.000s
I20260925 22:44:46.137662 15853 heartbeater.cc:503] Master 127.10.198.62:43407 was elected leader, sending a full tablet report...
I20260925 22:44:46.137822 16134 ts_tablet_manager.cc:1428] T 924cdcf581904c258fe67b288c5467d9 P 307e12871adf4718a24b3a48edad7b09: Time spent starting tablet: real 0.007s user 0.008s sys 0.000s
I20260925 22:44:46.138126 16132 ts_tablet_manager.cc:1428] T 924cdcf581904c258fe67b288c5467d9 P 1c1f35e4e0ab446e8ea73d158b74e17d: Time spent starting tablet: real 0.009s user 0.007s sys 0.000s
I20260925 22:44:46.138748 16115 heartbeater.cc:503] Master 127.10.198.62:43407 was elected leader, sending a full tablet report...
W20260925 22:44:46.170224 15854 tablet.cc:2369] T 924cdcf581904c258fe67b288c5467d9 P 11c3cbe22512415a955dc1b70010e3f0: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 22:44:46.234086 16138 raft_consensus.cc:482] T 924cdcf581904c258fe67b288c5467d9 P 1c1f35e4e0ab446e8ea73d158b74e17d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 22:44:46.234297 16138 raft_consensus.cc:504] T 924cdcf581904c258fe67b288c5467d9 P 1c1f35e4e0ab446e8ea73d158b74e17d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "307e12871adf4718a24b3a48edad7b09" member_type: VOTER last_known_addr { host: "127.10.198.3" port: 42461 } } peers { permanent_uuid: "11c3cbe22512415a955dc1b70010e3f0" member_type: VOTER last_known_addr { host: "127.10.198.1" port: 33801 } } peers { permanent_uuid: "1c1f35e4e0ab446e8ea73d158b74e17d" member_type: VOTER last_known_addr { host: "127.10.198.2" port: 34757 } }
I20260925 22:44:46.235502 16138 leader_election.cc:288] T 924cdcf581904c258fe67b288c5467d9 P 1c1f35e4e0ab446e8ea73d158b74e17d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 307e12871adf4718a24b3a48edad7b09 (127.10.198.3:42461), 11c3cbe22512415a955dc1b70010e3f0 (127.10.198.1:33801)
I20260925 22:44:46.237732 15986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:44:46.235782 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.198.1:33801 (local address 127.10.198.2:51715)
0925 22:44:46.235985 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0925 22:44:46.236006 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0925 22:44:46.236111 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:44:46.236507 (+ 396us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:44:46.236512 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:44:46.236528 (+ 16us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:44:46.237138 (+ 610us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 22:44:46.237141 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:44:46.237155 (+ 14us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 22:44:46.237185 (+ 30us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:44:46.237456 (+ 271us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:44:46.237459 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:44:46.237462 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 22:44:46.237550 (+ 88us) client_negotiation.cc:240] Negotiation successful
0925 22:44:46.237589 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":105}
I20260925 22:44:46.237733 16131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:44:46.235933 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.198.2:51715 (local address 127.10.198.1:33801)
0925 22:44:46.236101 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0925 22:44:46.236108 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 22:44:46.236131 (+ 23us) server_negotiation.cc:407] Connection header received
0925 22:44:46.236268 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:44:46.236272 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:44:46.236321 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:44:46.236418 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:44:46.237305 (+ 887us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:44:46.237309 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:44:46.237312 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 22:44:46.237333 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:44:46.237352 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 22:44:46.237361 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:44:46.237467 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0925 22:44:46.237547 (+ 80us) server_negotiation.cc:299] Negotiation successful
0925 22:44:46.237589 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260925 22:44:46.238116 16130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:44:46.235937 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.198.2:43785 (local address 127.10.198.3:42461)
0925 22:44:46.236092 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0925 22:44:46.236096 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:44:46.236695 (+ 599us) server_negotiation.cc:407] Connection header received
0925 22:44:46.236793 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:44:46.236798 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:44:46.236841 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:44:46.236939 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:44:46.237664 (+ 725us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:44:46.237667 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:44:46.237670 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 22:44:46.237691 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:44:46.237710 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 22:44:46.237718 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:44:46.237824 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0925 22:44:46.237929 (+ 105us) server_negotiation.cc:299] Negotiation successful
0925 22:44:46.237993 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260925 22:44:46.238461 16141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:44:46.235797 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.198.3:42461 (local address 127.10.198.2:43785)
0925 22:44:46.236553 (+ 756us) negotiation.cc:107] Waiting for socket to connect
0925 22:44:46.236570 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 22:44:46.236681 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:44:46.236983 (+ 302us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:44:46.236989 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:44:46.237008 (+ 19us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:44:46.237490 (+ 482us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 22:44:46.237492 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:44:46.237501 (+ 9us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 22:44:46.237529 (+ 28us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:44:46.237805 (+ 276us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:44:46.237809 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:44:46.237811 (+ 2us) client_negotiation.cc:769] Sending connection context
0925 22:44:46.238275 (+ 464us) client_negotiation.cc:240] Negotiation successful
0925 22:44:46.238321 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":687,"mutex_wait_us":194,"thread_start_us":191,"threads_started":1}
I20260925 22:44:46.238457 15807 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "924cdcf581904c258fe67b288c5467d9" candidate_uuid: "1c1f35e4e0ab446e8ea73d158b74e17d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "11c3cbe22512415a955dc1b70010e3f0" is_pre_election: true
I20260925 22:44:46.238828 15807 raft_consensus.cc:2454] T 924cdcf581904c258fe67b288c5467d9 P 11c3cbe22512415a955dc1b70010e3f0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c1f35e4e0ab446e8ea73d158b74e17d in term 0.
I20260925 22:44:46.239270 16069 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "924cdcf581904c258fe67b288c5467d9" candidate_uuid: "1c1f35e4e0ab446e8ea73d158b74e17d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "307e12871adf4718a24b3a48edad7b09" is_pre_election: true
I20260925 22:44:46.239377 15873 leader_election.cc:302] T 924cdcf581904c258fe67b288c5467d9 P 1c1f35e4e0ab446e8ea73d158b74e17d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 11c3cbe22512415a955dc1b70010e3f0, 1c1f35e4e0ab446e8ea73d158b74e17d; no voters:
I20260925 22:44:46.239621 16069 raft_consensus.cc:2454] T 924cdcf581904c258fe67b288c5467d9 P 307e12871adf4718a24b3a48edad7b09 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c1f35e4e0ab446e8ea73d158b74e17d in term 0.
I20260925 22:44:46.239708 16138 raft_consensus.cc:2781] T 924cdcf581904c258fe67b288c5467d9 P 1c1f35e4e0ab446e8ea73d158b74e17d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 22:44:46.239812 16138 raft_consensus.cc:482] T 924cdcf581904c258fe67b288c5467d9 P 1c1f35e4e0ab446e8ea73d158b74e17d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 22:44:46.239893 16138 raft_consensus.cc:3037] T 924cdcf581904c258fe67b288c5467d9 P 1c1f35e4e0ab446e8ea73d158b74e17d [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:44:46.241093 16138 raft_consensus.cc:504] T 924cdcf581904c258fe67b288c5467d9 P 1c1f35e4e0ab446e8ea73d158b74e17d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "307e12871adf4718a24b3a48edad7b09" member_type: VOTER last_known_addr { host: "127.10.198.3" port: 42461 } } peers { permanent_uuid: "11c3cbe22512415a955dc1b70010e3f0" member_type: VOTER last_known_addr { host: "127.10.198.1" port: 33801 } } peers { permanent_uuid: "1c1f35e4e0ab446e8ea73d158b74e17d" member_type: VOTER last_known_addr { host: "127.10.198.2" port: 34757 } }
I20260925 22:44:46.241515 16138 leader_election.cc:288] T 924cdcf581904c258fe67b288c5467d9 P 1c1f35e4e0ab446e8ea73d158b74e17d [CANDIDATE]: Term 1 election: Requested vote from peers 307e12871adf4718a24b3a48edad7b09 (127.10.198.3:42461), 11c3cbe22512415a955dc1b70010e3f0 (127.10.198.1:33801)
I20260925 22:44:46.241900 16069 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "924cdcf581904c258fe67b288c5467d9" candidate_uuid: "1c1f35e4e0ab446e8ea73d158b74e17d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "307e12871adf4718a24b3a48edad7b09"
I20260925 22:44:46.241987 15807 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "924cdcf581904c258fe67b288c5467d9" candidate_uuid: "1c1f35e4e0ab446e8ea73d158b74e17d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "11c3cbe22512415a955dc1b70010e3f0"
I20260925 22:44:46.242110 15807 raft_consensus.cc:3037] T 924cdcf581904c258fe67b288c5467d9 P 11c3cbe22512415a955dc1b70010e3f0 [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:44:46.242064 16069 raft_consensus.cc:3037] T 924cdcf581904c258fe67b288c5467d9 P 307e12871adf4718a24b3a48edad7b09 [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:44:46.243180 15807 raft_consensus.cc:2454] T 924cdcf581904c258fe67b288c5467d9 P 11c3cbe22512415a955dc1b70010e3f0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1c1f35e4e0ab446e8ea73d158b74e17d in term 1.
I20260925 22:44:46.243422 16069 raft_consensus.cc:2454] T 924cdcf581904c258fe67b288c5467d9 P 307e12871adf4718a24b3a48edad7b09 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1c1f35e4e0ab446e8ea73d158b74e17d in term 1.
I20260925 22:44:46.243584 15873 leader_election.cc:302] T 924cdcf581904c258fe67b288c5467d9 P 1c1f35e4e0ab446e8ea73d158b74e17d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 11c3cbe22512415a955dc1b70010e3f0, 1c1f35e4e0ab446e8ea73d158b74e17d; no voters:
I20260925 22:44:46.243821 16138 raft_consensus.cc:2781] T 924cdcf581904c258fe67b288c5467d9 P 1c1f35e4e0ab446e8ea73d158b74e17d [term 1 FOLLOWER]: Leader election won for term 1
I20260925 22:44:46.244078 16138 raft_consensus.cc:686] T 924cdcf581904c258fe67b288c5467d9 P 1c1f35e4e0ab446e8ea73d158b74e17d [term 1 LEADER]: Becoming Leader. State: Replica: 1c1f35e4e0ab446e8ea73d158b74e17d, State: Running, Role: LEADER
I20260925 22:44:46.244493 16138 consensus_queue.cc:237] T 924cdcf581904c258fe67b288c5467d9 P 1c1f35e4e0ab446e8ea73d158b74e17d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "307e12871adf4718a24b3a48edad7b09" member_type: VOTER last_known_addr { host: "127.10.198.3" port: 42461 } } peers { permanent_uuid: "11c3cbe22512415a955dc1b70010e3f0" member_type: VOTER last_known_addr { host: "127.10.198.1" port: 33801 } } peers { permanent_uuid: "1c1f35e4e0ab446e8ea73d158b74e17d" member_type: VOTER last_known_addr { host: "127.10.198.2" port: 34757 } }
I20260925 22:44:46.248564 15663 catalog_manager.cc:5495] T 924cdcf581904c258fe67b288c5467d9 P 1c1f35e4e0ab446e8ea73d158b74e17d reported cstate change: term changed from 0 to 1, leader changed from <none> to 1c1f35e4e0ab446e8ea73d158b74e17d (127.10.198.2). New cstate: current_term: 1 leader_uuid: "1c1f35e4e0ab446e8ea73d158b74e17d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "307e12871adf4718a24b3a48edad7b09" member_type: VOTER last_known_addr { host: "127.10.198.3" port: 42461 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "11c3cbe22512415a955dc1b70010e3f0" member_type: VOTER last_known_addr { host: "127.10.198.1" port: 33801 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1c1f35e4e0ab446e8ea73d158b74e17d" member_type: VOTER last_known_addr { host: "127.10.198.2" port: 34757 } health_report { overall_health: HEALTHY } } }
I20260925 22:44:46.267643 16129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:44:46.265896 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39762 (local address 127.10.198.2:34757)
0925 22:44:46.266054 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0925 22:44:46.266058 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:44:46.266073 (+ 15us) server_negotiation.cc:407] Connection header received
0925 22:44:46.266303 (+ 230us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:44:46.266307 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:44:46.266398 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:44:46.266493 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:44:46.267248 (+ 755us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:44:46.267250 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:44:46.267252 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 22:44:46.267264 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:44:46.267285 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 22:44:46.267294 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:44:46.267394 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0925 22:44:46.267492 (+ 98us) server_negotiation.cc:299] Negotiation successful
0925 22:44:46.267531 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260925 22:44:46.274794 16069 raft_consensus.cc:1261] T 924cdcf581904c258fe67b288c5467d9 P 307e12871adf4718a24b3a48edad7b09 [term 1 FOLLOWER]: Refusing update from remote peer 1c1f35e4e0ab446e8ea73d158b74e17d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 22:44:46.274794 15807 raft_consensus.cc:1261] T 924cdcf581904c258fe67b288c5467d9 P 11c3cbe22512415a955dc1b70010e3f0 [term 1 FOLLOWER]: Refusing update from remote peer 1c1f35e4e0ab446e8ea73d158b74e17d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 22:44:46.275712 16142 consensus_queue.cc:1035] T 924cdcf581904c258fe67b288c5467d9 P 1c1f35e4e0ab446e8ea73d158b74e17d [LEADER]: Connected to new peer: Peer: permanent_uuid: "307e12871adf4718a24b3a48edad7b09" member_type: VOTER last_known_addr { host: "127.10.198.3" port: 42461 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 22:44:46.275961 16138 consensus_queue.cc:1035] T 924cdcf581904c258fe67b288c5467d9 P 1c1f35e4e0ab446e8ea73d158b74e17d [LEADER]: Connected to new peer: Peer: permanent_uuid: "11c3cbe22512415a955dc1b70010e3f0" member_type: VOTER last_known_addr { host: "127.10.198.1" port: 33801 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 22:44:46.286072 16146 mvcc.cc:205] Tried to move back new op lower bound from 7333381268573011968 to 7333381268460761088. Current Snapshot: MvccSnapshot[applied={T|T < 7333381268573011968}]
I20260925 22:44:46.286839 16147 mvcc.cc:205] Tried to move back new op lower bound from 7333381268573011968 to 7333381268460761088. Current Snapshot: MvccSnapshot[applied={T|T < 7333381268573011968}]
I20260925 22:44:46.297370 15665 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41082:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 22:44:46.297587 15665 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41082:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 22:44:46.300381 15665 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e3469d866f62462fb807f2085654bd93: Sending DeleteTablet for 3 replicas of tablet 924cdcf581904c258fe67b288c5467d9
I20260925 22:44:46.301422 15787 tablet_service.cc:1514] Processing DeleteTablet for tablet 924cdcf581904c258fe67b288c5467d9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 22:44:46 UTC) from {username='slave'} at 127.0.0.1:38020
I20260925 22:44:46.301510 16049 tablet_service.cc:1514] Processing DeleteTablet for tablet 924cdcf581904c258fe67b288c5467d9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 22:44:46 UTC) from {username='slave'} at 127.0.0.1:37342
I20260925 22:44:46.301506 15917 tablet_service.cc:1514] Processing DeleteTablet for tablet 924cdcf581904c258fe67b288c5467d9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 22:44:46 UTC) from {username='slave'} at 127.0.0.1:39748
I20260925 22:44:46.302083 16157 tablet_replica.cc:330] T 924cdcf581904c258fe67b288c5467d9 P 1c1f35e4e0ab446e8ea73d158b74e17d: stopping tablet replica
I20260925 22:44:46.302397 16155 tablet_replica.cc:330] T 924cdcf581904c258fe67b288c5467d9 P 11c3cbe22512415a955dc1b70010e3f0: stopping tablet replica
I20260925 22:44:46.302510 16157 raft_consensus.cc:2229] T 924cdcf581904c258fe67b288c5467d9 P 1c1f35e4e0ab446e8ea73d158b74e17d [term 1 LEADER]: Raft consensus shutting down.
I20260925 22:44:46.302736 16155 raft_consensus.cc:2229] T 924cdcf581904c258fe67b288c5467d9 P 11c3cbe22512415a955dc1b70010e3f0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 22:44:46.303007 16157 raft_consensus.cc:2258] T 924cdcf581904c258fe67b288c5467d9 P 1c1f35e4e0ab446e8ea73d158b74e17d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 22:44:46.303104 16155 raft_consensus.cc:2258] T 924cdcf581904c258fe67b288c5467d9 P 11c3cbe22512415a955dc1b70010e3f0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 22:44:46.303361 11032 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrWXm8X/build/debug/bin/kudu with pid 15725
I20260925 22:44:46.303442 16156 tablet_replica.cc:330] T 924cdcf581904c258fe67b288c5467d9 P 307e12871adf4718a24b3a48edad7b09: stopping tablet replica
I20260925 22:44:46.303790 16156 raft_consensus.cc:2229] T 924cdcf581904c258fe67b288c5467d9 P 307e12871adf4718a24b3a48edad7b09 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 22:44:46.304159 16156 raft_consensus.cc:2258] T 924cdcf581904c258fe67b288c5467d9 P 307e12871adf4718a24b3a48edad7b09 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 22:44:46.306068 16156 ts_tablet_manager.cc:1905] T 924cdcf581904c258fe67b288c5467d9 P 307e12871adf4718a24b3a48edad7b09: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 22:44:46.307991 16157 ts_tablet_manager.cc:1905] T 924cdcf581904c258fe67b288c5467d9 P 1c1f35e4e0ab446e8ea73d158b74e17d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 22:44:46.309621 16156 ts_tablet_manager.cc:1918] T 924cdcf581904c258fe67b288c5467d9 P 307e12871adf4718a24b3a48edad7b09: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 22:44:46.309773 16156 log.cc:1196] T 924cdcf581904c258fe67b288c5467d9 P 307e12871adf4718a24b3a48edad7b09: Deleting WAL directory at /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790376259650642-11032-0/minicluster-data/ts-2/wal/wals/924cdcf581904c258fe67b288c5467d9
I20260925 22:44:46.310168 16156 ts_tablet_manager.cc:1939] T 924cdcf581904c258fe67b288c5467d9 P 307e12871adf4718a24b3a48edad7b09: Deleting consensus metadata
I20260925 22:44:46.311291 16157 ts_tablet_manager.cc:1918] T 924cdcf581904c258fe67b288c5467d9 P 1c1f35e4e0ab446e8ea73d158b74e17d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 22:44:46.311426 16157 log.cc:1196] T 924cdcf581904c258fe67b288c5467d9 P 1c1f35e4e0ab446e8ea73d158b74e17d: Deleting WAL directory at /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790376259650642-11032-0/minicluster-data/ts-1/wal/wals/924cdcf581904c258fe67b288c5467d9
I20260925 22:44:46.311834 16157 ts_tablet_manager.cc:1939] T 924cdcf581904c258fe67b288c5467d9 P 1c1f35e4e0ab446e8ea73d158b74e17d: Deleting consensus metadata
I20260925 22:44:46.311924 15652 catalog_manager.cc:4841] TS 307e12871adf4718a24b3a48edad7b09 (127.10.198.3:42461): tablet 924cdcf581904c258fe67b288c5467d9 (table test-table [id=4f508aef195b49fe945f3c6df7cb2728]) successfully deleted
I20260925 22:44:46.313244 15649 catalog_manager.cc:4841] TS 1c1f35e4e0ab446e8ea73d158b74e17d (127.10.198.2:34757): tablet 924cdcf581904c258fe67b288c5467d9 (table test-table [id=4f508aef195b49fe945f3c6df7cb2728]) successfully deleted
W20260925 22:44:46.313687 15651 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.10.198.1:33801 (error 108)
I20260925 22:44:46.313969 11032 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrWXm8X/build/debug/bin/kudu with pid 15856
I20260925 22:44:46.315677 16126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:44:46.315163 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.198.1:33801 (local address 127.0.0.1:38026)
0925 22:44:46.315452 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0925 22:44:46.315545 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.198.1:33801: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":126}
W20260925 22:44:46.315985 15651 catalog_manager.cc:4584] TS 11c3cbe22512415a955dc1b70010e3f0 (127.10.198.1:33801): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 924cdcf581904c258fe67b288c5467d9: Network error: Client connection negotiation failed: client connection to 127.10.198.1:33801: connect: Connection refused (error 111)
I20260925 22:44:46.320223 11032 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrWXm8X/build/debug/bin/kudu with pid 15987
I20260925 22:44:46.326262 11032 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrWXm8X/build/debug/bin/kudu with pid 15634
2026-09-25T22:44:46Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (1116 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260925 22:44:46.349146 11032 mini_oidc.cc:152] Starting JWKS server
I20260925 22:44:46.350862 11032 webserver.cc:460] Webserver started at https://127.0.0.1:45091/ using document root <none> and password file <none>
I20260925 22:44:46.351064 11032 mini_oidc.cc:189] Starting OIDC Discovery server
I20260925 22:44:46.351399 11032 webserver.cc:460] Webserver started at http://127.0.0.1:39117/ using document root <none> and password file <none>
2026-09-25T22:44:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T22:44:46Z Disabled control of system clock
I20260925 22:44:46.361532 11032 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrWXm8X/build/debug/bin/kudu
/tmp/dist-test-taskrWXm8X/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790376259650642-11032-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790376259650642-11032-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790376259650642-11032-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790376259650642-11032-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.198.62:32955
--webserver_interface=127.10.198.62
--webserver_port=0
--builtin_ntp_servers=127.10.198.20:41289
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.198.62:32955
--enable_jwt_token_auth=true
--jwks_url=https://localhost:45091/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790376259650642-11032-0/testchainca.cert with env {}
W20260925 22:44:46.481514 16164 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:44:46.482051 16164 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:44:46.482157 16164 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:44:46.486878 16164 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 22:44:46.487032 16164 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:44:46.487072 16164 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:44:46.487116 16164 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 22:44:46.487157 16164 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260925 22:44:46.487198 16164 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260925 22:44:46.487232 16164 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:45091/jwks.json
I20260925 22:44:46.493399 16164 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.198.20:41289
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790376259650642-11032-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790376259650642-11032-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.198.62:32955
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.198.62:32955
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:45091/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790376259650642-11032-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.198.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790376259650642-11032-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16164
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790376259650642-11032-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:09:16 UTC on ccycloud.quasar-afmhyw.root.comops.site
build id 4673379
I20260925 22:44:46.494699 16164 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:44:46.495929 16164 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:44:46.505679 16172 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:44:46.505609 16169 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:44:46.505813 16164 server_base.cc:1034] running on GCE node
W20260925 22:44:46.505609 16170 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:44:46.506819 16164 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:44:46.508353 16164 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:44:46.509609 16164 hybrid_clock.cc:648] HybridClock initialized: now 1790376286509591 us; error 42 us; skew 500 ppm
I20260925 22:44:46.512284 16164 webserver.cc:460] Webserver started at http://127.10.198.62:43383/ using document root <none> and password file <none>
I20260925 22:44:46.513466 16164 fs_manager.cc:359] Metadata directory not provided
I20260925 22:44:46.513592 16164 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:44:46.513863 16164 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:44:46.518164 16164 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790376259650642-11032-0/minicluster-data/master-0/data/instance:
uuid: "c8ea07995bc84689bd6c5c7f36029fa0"
format_stamp: "Formatted at 2026-09-25 22:44:46 on dist-test-slave-5vdr"
server_key: "74a6dfa8b03a7f639ee12376729d32d1"
server_key_iv: "45d40d27e7d6a3736878e07dc4e132df"
server_key_version: "encryptionkey@0"
I20260925 22:44:46.519409 16164 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790376259650642-11032-0/minicluster-data/master-0/wal/instance:
uuid: "c8ea07995bc84689bd6c5c7f36029fa0"
format_stamp: "Formatted at 2026-09-25 22:44:46 on dist-test-slave-5vdr"
server_key: "74a6dfa8b03a7f639ee12376729d32d1"
server_key_iv: "45d40d27e7d6a3736878e07dc4e132df"
server_key_version: "encryptionkey@0"
I20260925 22:44:46.526681 16164 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.009s sys 0.000s
I20260925 22:44:46.531155 16178 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:44:46.533192 16164 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260925 22:44:46.533377 16164 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790376259650642-11032-0/minicluster-data/master-0/data,/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790376259650642-11032-0/minicluster-data/master-0/wal
uuid: "c8ea07995bc84689bd6c5c7f36029fa0"
format_stamp: "Formatted at 2026-09-25 22:44:46 on dist-test-slave-5vdr"
server_key: "74a6dfa8b03a7f639ee12376729d32d1"
server_key_iv: "45d40d27e7d6a3736878e07dc4e132df"
server_key_version: "encryptionkey@0"
I20260925 22:44:46.533569 16164 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790376259650642-11032-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790376259650642-11032-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790376259650642-11032-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:44:46.550251 16164 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:44:46.550982 16164 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:44:46.551198 16164 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:44:46.559116 16164 rpc_server.cc:307] RPC server started. Bound to: 127.10.198.62:32955
I20260925 22:44:46.559115 16230 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.198.62:32955 every 8 connection(s)
I20260925 22:44:46.560382 16164 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790376259650642-11032-0/minicluster-data/master-0/data/info.pb
I20260925 22:44:46.564302 16231 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:44:46.569161 11032 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrWXm8X/build/debug/bin/kudu as pid 16164
I20260925 22:44:46.569291 11032 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790376259650642-11032-0/minicluster-data/master-0/wal/instance
I20260925 22:44:46.569621 11032 external_mini_cluster.cc:1428] Setting key 5e8cf5829a105549b4cb095c58b718fb
I20260925 22:44:46.571132 16231 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c8ea07995bc84689bd6c5c7f36029fa0: Bootstrap starting.
I20260925 22:44:46.573891 16231 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c8ea07995bc84689bd6c5c7f36029fa0: Neither blocks nor log segments found. Creating new log.
I20260925 22:44:46.574822 16231 log.cc:824] T 00000000000000000000000000000000 P c8ea07995bc84689bd6c5c7f36029fa0: Log is configured to *not* fsync() on all Append() calls
I20260925 22:44:46.577459 16231 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c8ea07995bc84689bd6c5c7f36029fa0: No bootstrap required, opened a new log
I20260925 22:44:46.581521 16234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:44:46.571263 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36776 (local address 127.10.198.62:32955)
0925 22:44:46.572115 (+ 852us) server_negotiation.cc:206] Beginning negotiation
0925 22:44:46.572132 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0925 22:44:46.572175 (+ 43us) server_negotiation.cc:407] Connection header received
0925 22:44:46.572935 (+ 760us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:44:46.572957 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:44:46.573424 (+ 467us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:44:46.573779 (+ 355us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:44:46.574290 (+ 511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:44:46.575414 (+ 1124us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:44:46.576546 (+ 1132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:44:46.577265 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:44:46.577440 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:44:46.578511 (+ 1071us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:44:46.578518 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:44:46.578538 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 22:44:46.579118 (+ 580us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:44:46.579213 (+ 95us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 22:44:46.579247 (+ 34us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:44:46.579360 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0925 22:44:46.579667 (+ 307us) server_negotiation.cc:299] Negotiation successful
0925 22:44:46.580102 (+ 435us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":327,"thread_start_us":156,"threads_started":1}
I20260925 22:44:46.582368 16231 raft_consensus.cc:348] T 00000000000000000000000000000000 P c8ea07995bc84689bd6c5c7f36029fa0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8ea07995bc84689bd6c5c7f36029fa0" member_type: VOTER last_known_addr { host: "127.10.198.62" port: 32955 } }
I20260925 22:44:46.582726 16231 raft_consensus.cc:374] T 00000000000000000000000000000000 P c8ea07995bc84689bd6c5c7f36029fa0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:44:46.582788 16231 raft_consensus.cc:729] T 00000000000000000000000000000000 P c8ea07995bc84689bd6c5c7f36029fa0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c8ea07995bc84689bd6c5c7f36029fa0, State: Initialized, Role: FOLLOWER
I20260925 22:44:46.583316 16231 consensus_queue.cc:260] T 00000000000000000000000000000000 P c8ea07995bc84689bd6c5c7f36029fa0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8ea07995bc84689bd6c5c7f36029fa0" member_type: VOTER last_known_addr { host: "127.10.198.62" port: 32955 } }
I20260925 22:44:46.583470 16231 raft_consensus.cc:388] T 00000000000000000000000000000000 P c8ea07995bc84689bd6c5c7f36029fa0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 22:44:46.583575 16231 raft_consensus.cc:482] T 00000000000000000000000000000000 P c8ea07995bc84689bd6c5c7f36029fa0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 22:44:46.583719 16231 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c8ea07995bc84689bd6c5c7f36029fa0 [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:44:46.585346 16231 raft_consensus.cc:504] T 00000000000000000000000000000000 P c8ea07995bc84689bd6c5c7f36029fa0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8ea07995bc84689bd6c5c7f36029fa0" member_type: VOTER last_known_addr { host: "127.10.198.62" port: 32955 } }
I20260925 22:44:46.585480 11032 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 22:44:46.585800 16231 leader_election.cc:302] T 00000000000000000000000000000000 P c8ea07995bc84689bd6c5c7f36029fa0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c8ea07995bc84689bd6c5c7f36029fa0; no voters:
I20260925 22:44:46.586297 16231 leader_election.cc:288] T 00000000000000000000000000000000 P c8ea07995bc84689bd6c5c7f36029fa0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 22:44:46.586422 16236 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c8ea07995bc84689bd6c5c7f36029fa0 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 22:44:46.586673 16236 raft_consensus.cc:686] T 00000000000000000000000000000000 P c8ea07995bc84689bd6c5c7f36029fa0 [term 1 LEADER]: Becoming Leader. State: Replica: c8ea07995bc84689bd6c5c7f36029fa0, State: Running, Role: LEADER
I20260925 22:44:46.587028 16236 consensus_queue.cc:237] T 00000000000000000000000000000000 P c8ea07995bc84689bd6c5c7f36029fa0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8ea07995bc84689bd6c5c7f36029fa0" member_type: VOTER last_known_addr { host: "127.10.198.62" port: 32955 } }
I20260925 22:44:46.587855 16231 sys_catalog.cc:564] T 00000000000000000000000000000000 P c8ea07995bc84689bd6c5c7f36029fa0 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 22:44:46.588922 16238 sys_catalog.cc:455] T 00000000000000000000000000000000 P c8ea07995bc84689bd6c5c7f36029fa0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c8ea07995bc84689bd6c5c7f36029fa0. Latest consensus state: current_term: 1 leader_uuid: "c8ea07995bc84689bd6c5c7f36029fa0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8ea07995bc84689bd6c5c7f36029fa0" member_type: VOTER last_known_addr { host: "127.10.198.62" port: 32955 } } }
I20260925 22:44:46.589128 16238 sys_catalog.cc:458] T 00000000000000000000000000000000 P c8ea07995bc84689bd6c5c7f36029fa0 [sys.catalog]: This master's current role is: LEADER
I20260925 22:44:46.589598 16237 sys_catalog.cc:455] T 00000000000000000000000000000000 P c8ea07995bc84689bd6c5c7f36029fa0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c8ea07995bc84689bd6c5c7f36029fa0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8ea07995bc84689bd6c5c7f36029fa0" member_type: VOTER last_known_addr { host: "127.10.198.62" port: 32955 } } }
I20260925 22:44:46.589751 16237 sys_catalog.cc:458] T 00000000000000000000000000000000 P c8ea07995bc84689bd6c5c7f36029fa0 [sys.catalog]: This master's current role is: LEADER
I20260925 22:44:46.590816 16245 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 22:44:46.594115 16245 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 22:44:46.601293 16245 catalog_manager.cc:1347] Generated new cluster ID: 6e8a1f7dff1d4d5fb2194f230fd51e67
I20260925 22:44:46.601421 16245 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 22:44:46.617085 16245 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 22:44:46.618211 16245 catalog_manager.cc:1504] Loading token signing keys...
I20260925 22:44:46.623790 16245 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c8ea07995bc84689bd6c5c7f36029fa0: Generated new TSK 0
I20260925 22:44:46.624629 16245 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260925 22:44:46.662225 16261 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260925 22:44:46.662377 16261 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260925 22:44:46.662875 16234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:44:46.638909 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36782 (local address 127.10.198.62:32955)
0925 22:44:46.639081 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0925 22:44:46.639087 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 22:44:46.639336 (+ 249us) server_negotiation.cc:407] Connection header received
0925 22:44:46.639629 (+ 293us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:44:46.639633 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:44:46.639708 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:44:46.639890 (+ 182us) server_negotiation.cc:226] Negotiated authn=JWT
0925 22:44:46.640313 (+ 423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:44:46.640895 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:44:46.641881 (+ 986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:44:46.642121 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:44:46.642242 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:44:46.642648 (+ 406us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0925 22:44:46.662330 (+ 19682us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45091/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0925 22:44:46.662709 (+ 379us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:36782: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45091/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":64}
W20260925 22:44:46.662990 16234 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:36782: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45091/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260925 22:44:46.664350 16260 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:44:46.638757 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.198.62:32955 (local address 127.0.0.1:36782)
0925 22:44:46.639232 (+ 475us) negotiation.cc:107] Waiting for socket to connect
0925 22:44:46.639247 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 22:44:46.639428 (+ 181us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:44:46.639954 (+ 526us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:44:46.639958 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:44:46.639967 (+ 9us) client_negotiation.cc:189] Negotiated authn=JWT
0925 22:44:46.640157 (+ 190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:44:46.640166 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:44:46.641055 (+ 889us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:44:46.641060 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:44:46.641735 (+ 675us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:44:46.641744 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:44:46.642252 (+ 508us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:44:46.642256 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:44:46.642359 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:44:46.642416 (+ 57us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0925 22:44:46.662726 (+ 20310us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45091/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0925 22:44:46.663020 (+ 294us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.10.198.62:32955: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45091/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":285,"thread_start_us":152,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.10.198.62:32955: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45091/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260925 22:44:46.665395 11032 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrWXm8X/build/debug/bin/kudu with pid 16164
2026-09-25T22:44:46Z chronyd exiting
I20260925 22:44:46.703610 11032 test_util.cc:175] -----------------------------------------------
I20260925 22:44:46.703717 11032 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790376259650642-11032-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (356 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260925 22:44:46.705252 11032 mini_oidc.cc:152] Starting JWKS server
I20260925 22:44:46.706383 11032 webserver.cc:460] Webserver started at https://127.0.0.1:44829/ using document root <none> and password file <none>
I20260925 22:44:46.706486 11032 mini_oidc.cc:189] Starting OIDC Discovery server
I20260925 22:44:46.706728 11032 webserver.cc:460] Webserver started at http://127.0.0.1:42817/ using document root <none> and password file <none>
2026-09-25T22:44:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T22:44:46Z Disabled control of system clock
I20260925 22:44:46.717069 11032 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrWXm8X/build/debug/bin/kudu
/tmp/dist-test-taskrWXm8X/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790376259650642-11032-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790376259650642-11032-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790376259650642-11032-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790376259650642-11032-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.198.62:45659
--webserver_interface=127.10.198.62
--webserver_port=0
--builtin_ntp_servers=127.10.198.20:46603
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.198.62:45659
--enable_jwt_token_auth=true
--jwks_url=https://localhost:44829/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790376259650642-11032-0/testchainca.cert with env {}
W20260925 22:44:46.838829 16267 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:44:46.839154 16267 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:44:46.839197 16267 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:44:46.843531 16267 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 22:44:46.843618 16267 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:44:46.843634 16267 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:44:46.843652 16267 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 22:44:46.843667 16267 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260925 22:44:46.843684 16267 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260925 22:44:46.843698 16267 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:44829/jwks.json
I20260925 22:44:46.849891 16267 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.198.20:46603
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790376259650642-11032-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790376259650642-11032-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.198.62:45659
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.198.62:45659
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:44829/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790376259650642-11032-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.198.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790376259650642-11032-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16267
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790376259650642-11032-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:09:16 UTC on ccycloud.quasar-afmhyw.root.comops.site
build id 4673379
I20260925 22:44:46.851140 16267 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:44:46.852299 16267 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:44:46.861410 16273 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:44:46.861392 16272 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:44:46.861393 16275 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:44:46.862104 16267 server_base.cc:1034] running on GCE node
I20260925 22:44:46.862752 16267 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:44:46.864158 16267 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:44:46.865366 16267 hybrid_clock.cc:648] HybridClock initialized: now 1790376286865323 us; error 66 us; skew 500 ppm
I20260925 22:44:46.867589 16267 webserver.cc:460] Webserver started at http://127.10.198.62:39693/ using document root <none> and password file <none>
I20260925 22:44:46.868324 16267 fs_manager.cc:359] Metadata directory not provided
I20260925 22:44:46.868431 16267 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:44:46.868685 16267 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:44:46.870956 16267 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790376259650642-11032-0/minicluster-data/master-0/data/instance:
uuid: "0d01860454bd4934bb1afac4c0d49943"
format_stamp: "Formatted at 2026-09-25 22:44:46 on dist-test-slave-5vdr"
server_key: "681ad35e908894d52a40f505259821c6"
server_key_iv: "00ce5ded962fd0693010f37262e07e1b"
server_key_version: "encryptionkey@0"
I20260925 22:44:46.871614 16267 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790376259650642-11032-0/minicluster-data/master-0/wal/instance:
uuid: "0d01860454bd4934bb1afac4c0d49943"
format_stamp: "Formatted at 2026-09-25 22:44:46 on dist-test-slave-5vdr"
server_key: "681ad35e908894d52a40f505259821c6"
server_key_iv: "00ce5ded962fd0693010f37262e07e1b"
server_key_version: "encryptionkey@0"
I20260925 22:44:46.876042 16267 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260925 22:44:46.879015 16281 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:44:46.880632 16267 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 22:44:46.880828 16267 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790376259650642-11032-0/minicluster-data/master-0/data,/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790376259650642-11032-0/minicluster-data/master-0/wal
uuid: "0d01860454bd4934bb1afac4c0d49943"
format_stamp: "Formatted at 2026-09-25 22:44:46 on dist-test-slave-5vdr"
server_key: "681ad35e908894d52a40f505259821c6"
server_key_iv: "00ce5ded962fd0693010f37262e07e1b"
server_key_version: "encryptionkey@0"
I20260925 22:44:46.880982 16267 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790376259650642-11032-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790376259650642-11032-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790376259650642-11032-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:44:46.909955 16267 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:44:46.910838 16267 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:44:46.911110 16267 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:44:46.919464 16267 rpc_server.cc:307] RPC server started. Bound to: 127.10.198.62:45659
I20260925 22:44:46.919466 16333 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.198.62:45659 every 8 connection(s)
I20260925 22:44:46.920702 16267 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790376259650642-11032-0/minicluster-data/master-0/data/info.pb
I20260925 22:44:46.924518 16334 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:44:46.924862 11032 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrWXm8X/build/debug/bin/kudu as pid 16267
I20260925 22:44:46.925004 11032 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790376259650642-11032-0/minicluster-data/master-0/wal/instance
I20260925 22:44:46.925300 11032 external_mini_cluster.cc:1428] Setting key 4230f974baa2beff006adf2f0fb20bec
I20260925 22:44:46.932677 16334 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0d01860454bd4934bb1afac4c0d49943: Bootstrap starting.
I20260925 22:44:46.935308 16334 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0d01860454bd4934bb1afac4c0d49943: Neither blocks nor log segments found. Creating new log.
I20260925 22:44:46.936072 16337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:44:46.926728 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37058 (local address 127.10.198.62:45659)
0925 22:44:46.927535 (+ 807us) server_negotiation.cc:206] Beginning negotiation
0925 22:44:46.927556 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0925 22:44:46.927598 (+ 42us) server_negotiation.cc:407] Connection header received
0925 22:44:46.928439 (+ 841us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:44:46.928461 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:44:46.929037 (+ 576us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:44:46.929381 (+ 344us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:44:46.929776 (+ 395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:44:46.930774 (+ 998us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:44:46.932078 (+ 1304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:44:46.932711 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:44:46.932935 (+ 224us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:44:46.933965 (+ 1030us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:44:46.933971 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:44:46.933986 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 22:44:46.934421 (+ 435us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:44:46.934489 (+ 68us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 22:44:46.934511 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:44:46.934632 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0925 22:44:46.934813 (+ 181us) server_negotiation.cc:299] Negotiation successful
0925 22:44:46.935111 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":421,"thread_start_us":273,"threads_started":1}
I20260925 22:44:46.936321 16334 log.cc:824] T 00000000000000000000000000000000 P 0d01860454bd4934bb1afac4c0d49943: Log is configured to *not* fsync() on all Append() calls
I20260925 22:44:46.938841 16334 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0d01860454bd4934bb1afac4c0d49943: No bootstrap required, opened a new log
I20260925 22:44:46.939339 11032 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 22:44:46.943050 16334 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0d01860454bd4934bb1afac4c0d49943 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d01860454bd4934bb1afac4c0d49943" member_type: VOTER last_known_addr { host: "127.10.198.62" port: 45659 } }
I20260925 22:44:46.943379 16334 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0d01860454bd4934bb1afac4c0d49943 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:44:46.943468 16334 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0d01860454bd4934bb1afac4c0d49943 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d01860454bd4934bb1afac4c0d49943, State: Initialized, Role: FOLLOWER
I20260925 22:44:46.944079 16334 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0d01860454bd4934bb1afac4c0d49943 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d01860454bd4934bb1afac4c0d49943" member_type: VOTER last_known_addr { host: "127.10.198.62" port: 45659 } }
I20260925 22:44:46.944272 16334 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0d01860454bd4934bb1afac4c0d49943 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 22:44:46.944348 16334 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0d01860454bd4934bb1afac4c0d49943 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 22:44:46.944464 16334 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0d01860454bd4934bb1afac4c0d49943 [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:44:46.945559 16334 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0d01860454bd4934bb1afac4c0d49943 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d01860454bd4934bb1afac4c0d49943" member_type: VOTER last_known_addr { host: "127.10.198.62" port: 45659 } }
I20260925 22:44:46.945992 16334 leader_election.cc:302] T 00000000000000000000000000000000 P 0d01860454bd4934bb1afac4c0d49943 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0d01860454bd4934bb1afac4c0d49943; no voters:
I20260925 22:44:46.946373 16334 leader_election.cc:288] T 00000000000000000000000000000000 P 0d01860454bd4934bb1afac4c0d49943 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 22:44:46.946872 16340 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0d01860454bd4934bb1afac4c0d49943 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d01860454bd4934bb1afac4c0d49943" member_type: VOTER last_known_addr { host: "127.10.198.62" port: 45659 } } }
I20260925 22:44:46.947093 16340 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0d01860454bd4934bb1afac4c0d49943 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 22:44:46.946947 16339 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0d01860454bd4934bb1afac4c0d49943 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 22:44:46.947409 16339 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0d01860454bd4934bb1afac4c0d49943 [term 1 LEADER]: Becoming Leader. State: Replica: 0d01860454bd4934bb1afac4c0d49943, State: Running, Role: LEADER
I20260925 22:44:46.947767 16339 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0d01860454bd4934bb1afac4c0d49943 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d01860454bd4934bb1afac4c0d49943" member_type: VOTER last_known_addr { host: "127.10.198.62" port: 45659 } }
I20260925 22:44:46.947997 16334 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0d01860454bd4934bb1afac4c0d49943 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 22:44:46.949643 16340 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0d01860454bd4934bb1afac4c0d49943 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0d01860454bd4934bb1afac4c0d49943. Latest consensus state: current_term: 1 leader_uuid: "0d01860454bd4934bb1afac4c0d49943" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d01860454bd4934bb1afac4c0d49943" member_type: VOTER last_known_addr { host: "127.10.198.62" port: 45659 } } }
I20260925 22:44:46.949786 16340 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0d01860454bd4934bb1afac4c0d49943 [sys.catalog]: This master's current role is: LEADER
W20260925 22:44:46.953419 16353 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 0d01860454bd4934bb1afac4c0d49943: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 22:44:46.953536 16353 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 22:44:46.953593 16348 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 22:44:46.954492 16348 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 22:44:46.961256 16348 catalog_manager.cc:1347] Generated new cluster ID: 7a9a6ef68d4044d5932322d4569472eb
I20260925 22:44:46.961354 16348 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 22:44:46.976480 16348 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 22:44:46.977849 16348 catalog_manager.cc:1504] Loading token signing keys...
I20260925 22:44:46.985445 16348 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0d01860454bd4934bb1afac4c0d49943: Generated new TSK 0
I20260925 22:44:46.986403 16348 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260925 22:44:47.025534 16363 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260925 22:44:47.025648 16363 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260925 22:44:47.026391 16337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:44:47.003226 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37066 (local address 127.10.198.62:45659)
0925 22:44:47.003386 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0925 22:44:47.003391 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:44:47.003669 (+ 278us) server_negotiation.cc:407] Connection header received
0925 22:44:47.003937 (+ 268us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:44:47.003939 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:44:47.003996 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:44:47.004134 (+ 138us) server_negotiation.cc:226] Negotiated authn=JWT
0925 22:44:47.004431 (+ 297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:44:47.005037 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:44:47.006004 (+ 967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:44:47.006258 (+ 254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:44:47.006381 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:44:47.006770 (+ 389us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0925 22:44:47.026000 (+ 19230us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44829/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0925 22:44:47.026271 (+ 271us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:37066: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44829/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":64}
W20260925 22:44:47.026497 16337 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:37066: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44829/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20260925 22:44:47.026683 16362 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:44:47.003120 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.198.62:45659 (local address 127.0.0.1:37066)
0925 22:44:47.003550 (+ 430us) negotiation.cc:107] Waiting for socket to connect
0925 22:44:47.003563 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 22:44:47.003726 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:44:47.004132 (+ 406us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:44:47.004136 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:44:47.004145 (+ 9us) client_negotiation.cc:189] Negotiated authn=JWT
0925 22:44:47.004297 (+ 152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:44:47.004303 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:44:47.005174 (+ 871us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:44:47.005178 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:44:47.005854 (+ 676us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:44:47.005863 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:44:47.006409 (+ 546us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:44:47.006414 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:44:47.006516 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:44:47.006544 (+ 28us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0925 22:44:47.026251 (+ 19707us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44829/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0925 22:44:47.026491 (+ 240us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.10.198.62:45659: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44829/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":355,"thread_start_us":140,"threads_started":1}
I20260925 22:44:47.027383 11032 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrWXm8X/build/debug/bin/kudu with pid 16267
2026-09-25T22:44:47Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (362 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260925 22:44:47.067421 11032 mini_oidc.cc:152] Starting JWKS server
I20260925 22:44:47.068423 11032 webserver.cc:460] Webserver started at https://127.0.0.1:40399/ using document root <none> and password file <none>
I20260925 22:44:47.068516 11032 mini_oidc.cc:189] Starting OIDC Discovery server
I20260925 22:44:47.068748 11032 webserver.cc:460] Webserver started at http://127.0.0.1:38053/ using document root <none> and password file <none>
2026-09-25T22:44:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T22:44:47Z Disabled control of system clock
I20260925 22:44:47.079361 11032 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrWXm8X/build/debug/bin/kudu
/tmp/dist-test-taskrWXm8X/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790376259650642-11032-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790376259650642-11032-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790376259650642-11032-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790376259650642-11032-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.198.62:45563
--webserver_interface=127.10.198.62
--webserver_port=0
--builtin_ntp_servers=127.10.198.20:36273
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.198.62:45563
--enable_jwt_token_auth=true
--jwks_url=https://localhost:40399/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260925 22:44:47.205523 16369 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:44:47.205962 16369 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:44:47.206012 16369 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:44:47.211027 16369 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 22:44:47.211140 16369 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:44:47.211156 16369 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:44:47.211175 16369 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 22:44:47.211191 16369 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260925 22:44:47.211208 16369 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260925 22:44:47.211221 16369 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:40399/jwks.json
I20260925 22:44:47.216830 16369 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.198.20:36273
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790376259650642-11032-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790376259650642-11032-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.198.62:45563
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.198.62:45563
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:40399/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790376259650642-11032-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.198.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16369
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790376259650642-11032-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:09:16 UTC on ccycloud.quasar-afmhyw.root.comops.site
build id 4673379
I20260925 22:44:47.218017 16369 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:44:47.219499 16369 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:44:47.229616 16374 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:44:47.229615 16377 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:44:47.229627 16375 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:44:47.230050 16369 server_base.cc:1034] running on GCE node
I20260925 22:44:47.230788 16369 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:44:47.232316 16369 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:44:47.233528 16369 hybrid_clock.cc:648] HybridClock initialized: now 1790376287233493 us; error 56 us; skew 500 ppm
I20260925 22:44:47.235790 16369 webserver.cc:460] Webserver started at http://127.10.198.62:42059/ using document root <none> and password file <none>
I20260925 22:44:47.236609 16369 fs_manager.cc:359] Metadata directory not provided
I20260925 22:44:47.236689 16369 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:44:47.236963 16369 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:44:47.239342 16369 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790376259650642-11032-0/minicluster-data/master-0/data/instance:
uuid: "19e1d678ab34446c878c4d0f3b77a7c6"
format_stamp: "Formatted at 2026-09-25 22:44:47 on dist-test-slave-5vdr"
server_key: "50ee1788ca46ae0da5bf40801570ea64"
server_key_iv: "e53dc8281d7e2d3e1eb2976375d8d31c"
server_key_version: "encryptionkey@0"
I20260925 22:44:47.240084 16369 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790376259650642-11032-0/minicluster-data/master-0/wal/instance:
uuid: "19e1d678ab34446c878c4d0f3b77a7c6"
format_stamp: "Formatted at 2026-09-25 22:44:47 on dist-test-slave-5vdr"
server_key: "50ee1788ca46ae0da5bf40801570ea64"
server_key_iv: "e53dc8281d7e2d3e1eb2976375d8d31c"
server_key_version: "encryptionkey@0"
I20260925 22:44:47.244560 16369 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260925 22:44:47.247663 16383 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:44:47.249358 16369 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260925 22:44:47.249603 16369 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790376259650642-11032-0/minicluster-data/master-0/data,/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790376259650642-11032-0/minicluster-data/master-0/wal
uuid: "19e1d678ab34446c878c4d0f3b77a7c6"
format_stamp: "Formatted at 2026-09-25 22:44:47 on dist-test-slave-5vdr"
server_key: "50ee1788ca46ae0da5bf40801570ea64"
server_key_iv: "e53dc8281d7e2d3e1eb2976375d8d31c"
server_key_version: "encryptionkey@0"
I20260925 22:44:47.249815 16369 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790376259650642-11032-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790376259650642-11032-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790376259650642-11032-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:44:47.278398 16369 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:44:47.279196 16369 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:44:47.279422 16369 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:44:47.287235 16369 rpc_server.cc:307] RPC server started. Bound to: 127.10.198.62:45563
I20260925 22:44:47.287379 16435 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.198.62:45563 every 8 connection(s)
I20260925 22:44:47.288406 16369 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790376259650642-11032-0/minicluster-data/master-0/data/info.pb
I20260925 22:44:47.292026 16436 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:44:47.297670 11032 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrWXm8X/build/debug/bin/kudu as pid 16369
I20260925 22:44:47.297849 11032 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790376259650642-11032-0/minicluster-data/master-0/wal/instance
I20260925 22:44:47.298175 11032 external_mini_cluster.cc:1428] Setting key 7ac43da2e06c84278f956aaa3f5ac04e
I20260925 22:44:47.298404 16436 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 19e1d678ab34446c878c4d0f3b77a7c6: Bootstrap starting.
I20260925 22:44:47.301527 16436 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 19e1d678ab34446c878c4d0f3b77a7c6: Neither blocks nor log segments found. Creating new log.
I20260925 22:44:47.302538 16436 log.cc:824] T 00000000000000000000000000000000 P 19e1d678ab34446c878c4d0f3b77a7c6: Log is configured to *not* fsync() on all Append() calls
I20260925 22:44:47.305207 16436 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 19e1d678ab34446c878c4d0f3b77a7c6: No bootstrap required, opened a new log
I20260925 22:44:47.310142 16439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:44:47.299680 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39792 (local address 127.10.198.62:45563)
0925 22:44:47.300520 (+ 840us) server_negotiation.cc:206] Beginning negotiation
0925 22:44:47.300542 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0925 22:44:47.300592 (+ 50us) server_negotiation.cc:407] Connection header received
0925 22:44:47.301574 (+ 982us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:44:47.301602 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:44:47.302197 (+ 595us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:44:47.302648 (+ 451us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:44:47.303152 (+ 504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:44:47.304592 (+ 1440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:44:47.305693 (+ 1101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:44:47.306434 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:44:47.306621 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:44:47.307586 (+ 965us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:44:47.307594 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:44:47.307617 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 22:44:47.308230 (+ 613us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:44:47.308314 (+ 84us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 22:44:47.308343 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:44:47.308452 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0925 22:44:47.308590 (+ 138us) server_negotiation.cc:299] Negotiation successful
0925 22:44:47.308918 (+ 328us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":359,"thread_start_us":172,"threads_started":1}
I20260925 22:44:47.310472 16436 raft_consensus.cc:348] T 00000000000000000000000000000000 P 19e1d678ab34446c878c4d0f3b77a7c6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "19e1d678ab34446c878c4d0f3b77a7c6" member_type: VOTER last_known_addr { host: "127.10.198.62" port: 45563 } }
I20260925 22:44:47.310760 16436 raft_consensus.cc:374] T 00000000000000000000000000000000 P 19e1d678ab34446c878c4d0f3b77a7c6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:44:47.310844 16436 raft_consensus.cc:729] T 00000000000000000000000000000000 P 19e1d678ab34446c878c4d0f3b77a7c6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 19e1d678ab34446c878c4d0f3b77a7c6, State: Initialized, Role: FOLLOWER
I20260925 22:44:47.311306 16436 consensus_queue.cc:260] T 00000000000000000000000000000000 P 19e1d678ab34446c878c4d0f3b77a7c6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "19e1d678ab34446c878c4d0f3b77a7c6" member_type: VOTER last_known_addr { host: "127.10.198.62" port: 45563 } }
I20260925 22:44:47.311470 16436 raft_consensus.cc:388] T 00000000000000000000000000000000 P 19e1d678ab34446c878c4d0f3b77a7c6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 22:44:47.311564 16436 raft_consensus.cc:482] T 00000000000000000000000000000000 P 19e1d678ab34446c878c4d0f3b77a7c6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 22:44:47.311676 16436 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 19e1d678ab34446c878c4d0f3b77a7c6 [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:44:47.312888 16436 raft_consensus.cc:504] T 00000000000000000000000000000000 P 19e1d678ab34446c878c4d0f3b77a7c6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "19e1d678ab34446c878c4d0f3b77a7c6" member_type: VOTER last_known_addr { host: "127.10.198.62" port: 45563 } }
I20260925 22:44:47.313288 16436 leader_election.cc:302] T 00000000000000000000000000000000 P 19e1d678ab34446c878c4d0f3b77a7c6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 19e1d678ab34446c878c4d0f3b77a7c6; no voters:
I20260925 22:44:47.313661 11032 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 22:44:47.313674 16436 leader_election.cc:288] T 00000000000000000000000000000000 P 19e1d678ab34446c878c4d0f3b77a7c6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 22:44:47.313800 16441 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 19e1d678ab34446c878c4d0f3b77a7c6 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 22:44:47.314083 16441 raft_consensus.cc:686] T 00000000000000000000000000000000 P 19e1d678ab34446c878c4d0f3b77a7c6 [term 1 LEADER]: Becoming Leader. State: Replica: 19e1d678ab34446c878c4d0f3b77a7c6, State: Running, Role: LEADER
I20260925 22:44:47.314443 16441 consensus_queue.cc:237] T 00000000000000000000000000000000 P 19e1d678ab34446c878c4d0f3b77a7c6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "19e1d678ab34446c878c4d0f3b77a7c6" member_type: VOTER last_known_addr { host: "127.10.198.62" port: 45563 } }
I20260925 22:44:47.315174 16436 sys_catalog.cc:564] T 00000000000000000000000000000000 P 19e1d678ab34446c878c4d0f3b77a7c6 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 22:44:47.316418 16442 sys_catalog.cc:455] T 00000000000000000000000000000000 P 19e1d678ab34446c878c4d0f3b77a7c6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "19e1d678ab34446c878c4d0f3b77a7c6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "19e1d678ab34446c878c4d0f3b77a7c6" member_type: VOTER last_known_addr { host: "127.10.198.62" port: 45563 } } }
I20260925 22:44:47.316604 16442 sys_catalog.cc:458] T 00000000000000000000000000000000 P 19e1d678ab34446c878c4d0f3b77a7c6 [sys.catalog]: This master's current role is: LEADER
I20260925 22:44:47.316927 16443 sys_catalog.cc:455] T 00000000000000000000000000000000 P 19e1d678ab34446c878c4d0f3b77a7c6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 19e1d678ab34446c878c4d0f3b77a7c6. Latest consensus state: current_term: 1 leader_uuid: "19e1d678ab34446c878c4d0f3b77a7c6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "19e1d678ab34446c878c4d0f3b77a7c6" member_type: VOTER last_known_addr { host: "127.10.198.62" port: 45563 } } }
I20260925 22:44:47.317030 16443 sys_catalog.cc:458] T 00000000000000000000000000000000 P 19e1d678ab34446c878c4d0f3b77a7c6 [sys.catalog]: This master's current role is: LEADER
I20260925 22:44:47.317162 16450 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 22:44:47.320709 16450 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 22:44:47.326826 16450 catalog_manager.cc:1347] Generated new cluster ID: 6b58d65a079e4fc4b0e1cb6e052e26a7
I20260925 22:44:47.326922 16450 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 22:44:47.354599 16450 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 22:44:47.356019 16450 catalog_manager.cc:1504] Loading token signing keys...
I20260925 22:44:47.367179 16450 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 19e1d678ab34446c878c4d0f3b77a7c6: Generated new TSK 0
I20260925 22:44:47.368153 16450 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 22:44:47.389683 16439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:44:47.383696 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39796 (local address 127.10.198.62:45563)
0925 22:44:47.383879 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0925 22:44:47.383885 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 22:44:47.384008 (+ 123us) server_negotiation.cc:407] Connection header received
0925 22:44:47.384178 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:44:47.384182 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:44:47.384254 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:44:47.384395 (+ 141us) server_negotiation.cc:226] Negotiated authn=JWT
0925 22:44:47.384634 (+ 239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:44:47.385546 (+ 912us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:44:47.386526 (+ 980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:44:47.386889 (+ 363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:44:47.387016 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:44:47.387324 (+ 308us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0925 22:44:47.389149 (+ 1825us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40399/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0925 22:44:47.389498 (+ 349us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:39796: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40399/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":73}
W20260925 22:44:47.389778 16439 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:39796: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40399/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260925 22:44:47.391224 16465 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:44:47.383563 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.198.62:45563 (local address 127.0.0.1:39796)
0925 22:44:47.383892 (+ 329us) negotiation.cc:107] Waiting for socket to connect
0925 22:44:47.383909 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 22:44:47.384029 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:44:47.384408 (+ 379us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:44:47.384410 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:44:47.384416 (+ 6us) client_negotiation.cc:189] Negotiated authn=JWT
0925 22:44:47.384522 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:44:47.384526 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:44:47.385708 (+ 1182us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:44:47.385711 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:44:47.386382 (+ 671us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:44:47.386390 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:44:47.387021 (+ 631us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:44:47.387024 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:44:47.387099 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:44:47.387125 (+ 26us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0925 22:44:47.389397 (+ 2272us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40399/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0925 22:44:47.391017 (+ 1620us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.10.198.62:45563: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40399/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":208,"mutex_wait_us":1358,"thread_start_us":117,"threads_started":1}
I20260925 22:44:47.391937 11032 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrWXm8X/build/debug/bin/kudu with pid 16369
E20260925 22:44:47.402887 16466 webserver.cc:533] Webserver: sslize failed:
E20260925 22:44:47.403012 16466 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
2026-09-25T22:44:47Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (365 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-09-25T22:44:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T22:44:47Z Disabled control of system clock
I20260925 22:44:47.442312 11032 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrWXm8X/build/debug/bin/kudu
/tmp/dist-test-taskrWXm8X/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790376259650642-11032-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790376259650642-11032-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790376259650642-11032-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790376259650642-11032-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.198.62:44767
--webserver_interface=127.10.198.62
--webserver_port=0
--builtin_ntp_servers=127.10.198.20:33803
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.198.62:44767
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790376259650642-11032-0/testchainca.cert with env {}
W20260925 22:44:47.567576 16470 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:44:47.567968 16470 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:44:47.568018 16470 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:44:47.572340 16470 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 22:44:47.572438 16470 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:44:47.572456 16470 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:44:47.572477 16470 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 22:44:47.572496 16470 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260925 22:44:47.572516 16470 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260925 22:44:47.572533 16470 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260925 22:44:47.578456 16470 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.198.20:33803
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790376259650642-11032-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790376259650642-11032-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.198.62:44767
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.198.62:44767
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790376259650642-11032-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.198.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790376259650642-11032-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16470
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790376259650642-11032-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:09:16 UTC on ccycloud.quasar-afmhyw.root.comops.site
build id 4673379
I20260925 22:44:47.579646 16470 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:44:47.580746 16470 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:44:47.590610 16476 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:44:47.590613 16478 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:44:47.590610 16475 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:44:47.590667 16470 server_base.cc:1034] running on GCE node
I20260925 22:44:47.591670 16470 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:44:47.593050 16470 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:44:47.594257 16470 hybrid_clock.cc:648] HybridClock initialized: now 1790376287594225 us; error 54 us; skew 500 ppm
I20260925 22:44:47.596598 16470 webserver.cc:460] Webserver started at http://127.10.198.62:37161/ using document root <none> and password file <none>
I20260925 22:44:47.597288 16470 fs_manager.cc:359] Metadata directory not provided
I20260925 22:44:47.597364 16470 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:44:47.597622 16470 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:44:47.599998 16470 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790376259650642-11032-0/minicluster-data/master-0/data/instance:
uuid: "88f9ccfcb6f3473d9a37d250afe212ba"
format_stamp: "Formatted at 2026-09-25 22:44:47 on dist-test-slave-5vdr"
server_key: "64462741802b4f6dc5755733f4d89698"
server_key_iv: "e95be0f685527a60a2f0a3c26deccd0a"
server_key_version: "encryptionkey@0"
I20260925 22:44:47.600943 16470 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790376259650642-11032-0/minicluster-data/master-0/wal/instance:
uuid: "88f9ccfcb6f3473d9a37d250afe212ba"
format_stamp: "Formatted at 2026-09-25 22:44:47 on dist-test-slave-5vdr"
server_key: "64462741802b4f6dc5755733f4d89698"
server_key_iv: "e95be0f685527a60a2f0a3c26deccd0a"
server_key_version: "encryptionkey@0"
I20260925 22:44:47.605620 16470 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260925 22:44:47.608809 16484 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:44:47.610494 16470 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260925 22:44:47.610711 16470 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790376259650642-11032-0/minicluster-data/master-0/data,/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790376259650642-11032-0/minicluster-data/master-0/wal
uuid: "88f9ccfcb6f3473d9a37d250afe212ba"
format_stamp: "Formatted at 2026-09-25 22:44:47 on dist-test-slave-5vdr"
server_key: "64462741802b4f6dc5755733f4d89698"
server_key_iv: "e95be0f685527a60a2f0a3c26deccd0a"
server_key_version: "encryptionkey@0"
I20260925 22:44:47.610864 16470 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790376259650642-11032-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790376259650642-11032-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790376259650642-11032-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:44:47.629431 16470 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:44:47.630357 16470 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:44:47.630600 16470 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:44:47.638425 16470 rpc_server.cc:307] RPC server started. Bound to: 127.10.198.62:44767
I20260925 22:44:47.638451 16536 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.198.62:44767 every 8 connection(s)
I20260925 22:44:47.639570 16470 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790376259650642-11032-0/minicluster-data/master-0/data/info.pb
I20260925 22:44:47.640100 11032 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrWXm8X/build/debug/bin/kudu as pid 16470
I20260925 22:44:47.640288 11032 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790376259650642-11032-0/minicluster-data/master-0/wal/instance
I20260925 22:44:47.640767 11032 external_mini_cluster.cc:1428] Setting key 4e6c0d6baa016547ef5f7d19def2bcb2
I20260925 22:44:47.643672 16537 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:44:47.651085 16537 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 88f9ccfcb6f3473d9a37d250afe212ba: Bootstrap starting.
I20260925 22:44:47.651805 16540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:44:47.642427 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51548 (local address 127.10.198.62:44767)
0925 22:44:47.643189 (+ 762us) server_negotiation.cc:206] Beginning negotiation
0925 22:44:47.643209 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0925 22:44:47.643256 (+ 47us) server_negotiation.cc:407] Connection header received
0925 22:44:47.644102 (+ 846us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:44:47.644125 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:44:47.644595 (+ 470us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:44:47.644963 (+ 368us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:44:47.645395 (+ 432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:44:47.646446 (+ 1051us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:44:47.647754 (+ 1308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:44:47.648391 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:44:47.648554 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:44:47.649613 (+ 1059us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:44:47.649619 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:44:47.649634 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 22:44:47.650096 (+ 462us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:44:47.650163 (+ 67us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 22:44:47.650184 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:44:47.650273 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0925 22:44:47.650468 (+ 195us) server_negotiation.cc:299] Negotiation successful
0925 22:44:47.650830 (+ 362us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":340,"thread_start_us":174,"threads_started":1}
I20260925 22:44:47.653811 16537 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 88f9ccfcb6f3473d9a37d250afe212ba: Neither blocks nor log segments found. Creating new log.
I20260925 22:44:47.654899 16537 log.cc:824] T 00000000000000000000000000000000 P 88f9ccfcb6f3473d9a37d250afe212ba: Log is configured to *not* fsync() on all Append() calls
I20260925 22:44:47.655169 11032 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 22:44:47.657554 16537 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 88f9ccfcb6f3473d9a37d250afe212ba: No bootstrap required, opened a new log
I20260925 22:44:47.662281 16537 raft_consensus.cc:348] T 00000000000000000000000000000000 P 88f9ccfcb6f3473d9a37d250afe212ba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88f9ccfcb6f3473d9a37d250afe212ba" member_type: VOTER last_known_addr { host: "127.10.198.62" port: 44767 } }
I20260925 22:44:47.662650 16537 raft_consensus.cc:374] T 00000000000000000000000000000000 P 88f9ccfcb6f3473d9a37d250afe212ba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:44:47.662729 16537 raft_consensus.cc:729] T 00000000000000000000000000000000 P 88f9ccfcb6f3473d9a37d250afe212ba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 88f9ccfcb6f3473d9a37d250afe212ba, State: Initialized, Role: FOLLOWER
I20260925 22:44:47.663337 16537 consensus_queue.cc:260] T 00000000000000000000000000000000 P 88f9ccfcb6f3473d9a37d250afe212ba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88f9ccfcb6f3473d9a37d250afe212ba" member_type: VOTER last_known_addr { host: "127.10.198.62" port: 44767 } }
I20260925 22:44:47.663589 16537 raft_consensus.cc:388] T 00000000000000000000000000000000 P 88f9ccfcb6f3473d9a37d250afe212ba [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 22:44:47.663707 16537 raft_consensus.cc:482] T 00000000000000000000000000000000 P 88f9ccfcb6f3473d9a37d250afe212ba [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 22:44:47.663897 16537 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 88f9ccfcb6f3473d9a37d250afe212ba [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:44:47.665022 16537 raft_consensus.cc:504] T 00000000000000000000000000000000 P 88f9ccfcb6f3473d9a37d250afe212ba [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88f9ccfcb6f3473d9a37d250afe212ba" member_type: VOTER last_known_addr { host: "127.10.198.62" port: 44767 } }
I20260925 22:44:47.665475 16537 leader_election.cc:302] T 00000000000000000000000000000000 P 88f9ccfcb6f3473d9a37d250afe212ba [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 88f9ccfcb6f3473d9a37d250afe212ba; no voters:
I20260925 22:44:47.665875 16537 leader_election.cc:288] T 00000000000000000000000000000000 P 88f9ccfcb6f3473d9a37d250afe212ba [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 22:44:47.666031 16542 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 88f9ccfcb6f3473d9a37d250afe212ba [term 1 FOLLOWER]: Leader election won for term 1
I20260925 22:44:47.666343 16542 raft_consensus.cc:686] T 00000000000000000000000000000000 P 88f9ccfcb6f3473d9a37d250afe212ba [term 1 LEADER]: Becoming Leader. State: Replica: 88f9ccfcb6f3473d9a37d250afe212ba, State: Running, Role: LEADER
I20260925 22:44:47.666702 16542 consensus_queue.cc:237] T 00000000000000000000000000000000 P 88f9ccfcb6f3473d9a37d250afe212ba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88f9ccfcb6f3473d9a37d250afe212ba" member_type: VOTER last_known_addr { host: "127.10.198.62" port: 44767 } }
I20260925 22:44:47.667339 16537 sys_catalog.cc:564] T 00000000000000000000000000000000 P 88f9ccfcb6f3473d9a37d250afe212ba [sys.catalog]: configured and running, proceeding with master startup.
I20260925 22:44:47.668823 16544 sys_catalog.cc:455] T 00000000000000000000000000000000 P 88f9ccfcb6f3473d9a37d250afe212ba [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88f9ccfcb6f3473d9a37d250afe212ba. Latest consensus state: current_term: 1 leader_uuid: "88f9ccfcb6f3473d9a37d250afe212ba" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88f9ccfcb6f3473d9a37d250afe212ba" member_type: VOTER last_known_addr { host: "127.10.198.62" port: 44767 } } }
I20260925 22:44:47.668848 16543 sys_catalog.cc:455] T 00000000000000000000000000000000 P 88f9ccfcb6f3473d9a37d250afe212ba [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "88f9ccfcb6f3473d9a37d250afe212ba" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88f9ccfcb6f3473d9a37d250afe212ba" member_type: VOTER last_known_addr { host: "127.10.198.62" port: 44767 } } }
I20260925 22:44:47.668987 16544 sys_catalog.cc:458] T 00000000000000000000000000000000 P 88f9ccfcb6f3473d9a37d250afe212ba [sys.catalog]: This master's current role is: LEADER
I20260925 22:44:47.668992 16543 sys_catalog.cc:458] T 00000000000000000000000000000000 P 88f9ccfcb6f3473d9a37d250afe212ba [sys.catalog]: This master's current role is: LEADER
I20260925 22:44:47.669399 16551 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 22:44:47.672930 16551 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 22:44:47.679615 16551 catalog_manager.cc:1347] Generated new cluster ID: b1b405ecf3c446b2b7fc94af12ec53e9
I20260925 22:44:47.679718 16551 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 22:44:47.704576 16551 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 22:44:47.706087 16551 catalog_manager.cc:1504] Loading token signing keys...
I20260925 22:44:47.715571 16551 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 88f9ccfcb6f3473d9a37d250afe212ba: Generated new TSK 0
I20260925 22:44:47.716603 16551 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 22:44:47.778455 16540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:44:47.728255 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51554 (local address 127.10.198.62:44767)
0925 22:44:47.728418 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0925 22:44:47.728423 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:44:47.728647 (+ 224us) server_negotiation.cc:407] Connection header received
0925 22:44:47.728828 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:44:47.728831 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:44:47.728884 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:44:47.728986 (+ 102us) server_negotiation.cc:226] Negotiated authn=JWT
0925 22:44:47.729280 (+ 294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:44:47.729848 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:44:47.731188 (+ 1340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:44:47.731416 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:44:47.731515 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:44:47.731989 (+ 474us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0925 22:44:47.777878 (+ 45889us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0925 22:44:47.778267 (+ 389us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:51554: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":65}
W20260925 22:44:47.778558 16540 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:51554: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260925 22:44:47.778529 16566 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:44:47.728175 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.198.62:44767 (local address 127.0.0.1:51554)
0925 22:44:47.728538 (+ 363us) negotiation.cc:107] Waiting for socket to connect
0925 22:44:47.728555 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 22:44:47.728702 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:44:47.729015 (+ 313us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:44:47.729018 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:44:47.729027 (+ 9us) client_negotiation.cc:189] Negotiated authn=JWT
0925 22:44:47.729150 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:44:47.729157 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:44:47.729983 (+ 826us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:44:47.729986 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:44:47.731019 (+ 1033us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:44:47.731031 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:44:47.731565 (+ 534us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:44:47.731569 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:44:47.731672 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:44:47.731716 (+ 44us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0925 22:44:47.778188 (+ 46472us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0925 22:44:47.778376 (+ 188us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.10.198.62:44767: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":271,"thread_start_us":143,"threads_started":1}
I20260925 22:44:47.779282 11032 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrWXm8X/build/debug/bin/kudu with pid 16470
2026-09-25T22:44:47Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (367 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (133 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (133 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790376259650642-11032-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 22:44:48 dist-test-slave-5vdr krb5kdc[16578](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 22:44:48 dist-test-slave-5vdr krb5kdc[16578](info): set up 2 sockets
Sep 25 22:44:48 dist-test-slave-5vdr krb5kdc[16578](info): commencing operation
krb5kdc: starting...
W20260925 22:44:49.108496 11032 mini_kdc.cc:121] Time spent starting KDC: real 1.041s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790376259650642-11032-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790376259650642-11032-0/krb5kdc/test-user.keytab.
Sep 25 22:44:49 dist-test-slave-5vdr krb5kdc[16578](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376289, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T22:44:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T22:44:49Z Disabled control of system clock
WARNING: no policy specified for kudu/127.10.198.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.198.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.198.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790376259650642-11032-0/krb5kdc/kudu_127.10.198.62.keytab.
Entry for principal kudu/127.10.198.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790376259650642-11032-0/krb5kdc/kudu_127.10.198.62.keytab.
I20260925 22:44:49.244160 11032 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrWXm8X/build/debug/bin/kudu
/tmp/dist-test-taskrWXm8X/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790376259650642-11032-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790376259650642-11032-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790376259650642-11032-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790376259650642-11032-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.198.62:45183
--webserver_interface=127.10.198.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790376259650642-11032-0/krb5kdc/kudu_127.10.198.62.keytab
--principal=kudu/127.10.198.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.198.20:33605
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.198.62:45183
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790376259650642-11032-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790376259650642-11032-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790376259650642-11032-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:44:49.366487 16591 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:44:49.366920 16591 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:44:49.366969 16591 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:44:49.371423 16591 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 22:44:49.371523 16591 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:44:49.371548 16591 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:44:49.371568 16591 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 22:44:49.371587 16591 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 22:44:49.377635 16591 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.198.20:33605
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790376259650642-11032-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790376259650642-11032-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.198.62:45183
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790376259650642-11032-0/krb5kdc/kudu_127.10.198.62.keytab
--principal=kudu/127.10.198.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.198.62:45183
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790376259650642-11032-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.198.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16591
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790376259650642-11032-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:09:16 UTC on ccycloud.quasar-afmhyw.root.comops.site
build id 4673379
I20260925 22:44:49.378844 16591 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:44:49.380033 16591 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:44:49.390002 16597 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:44:49.389945 16599 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:44:49.389945 16596 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:44:49.390664 16591 server_base.cc:1034] running on GCE node
I20260925 22:44:49.391333 16591 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:44:49.392781 16591 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:44:49.394011 16591 hybrid_clock.cc:648] HybridClock initialized: now 1790376289393992 us; error 41 us; skew 500 ppm
Sep 25 22:44:49 dist-test-slave-5vdr krb5kdc[16578](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376289, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.198.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:44:49.403420 16591 init.cc:377] Logged in from keytab as kudu/127.10.198.62@KRBTEST.COM (short username kudu)
I20260925 22:44:49.404683 16591 webserver.cc:460] Webserver started at http://127.10.198.62:37735/ using document root <none> and password file <none>
I20260925 22:44:49.405314 16591 fs_manager.cc:359] Metadata directory not provided
I20260925 22:44:49.405407 16591 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:44:49.405613 16591 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:44:49.408160 16591 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790376259650642-11032-0/minicluster-data/master-0/data/instance:
uuid: "850b7724fd5b4d77857edd66e386b90e"
format_stamp: "Formatted at 2026-09-25 22:44:49 on dist-test-slave-5vdr"
server_key: "040384d8dca10f4046a39a49011f9c92"
server_key_iv: "49889fe9af6c2cfb61f3480da9a79be0"
server_key_version: "encryptionkey@0"
I20260925 22:44:49.408812 16591 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790376259650642-11032-0/minicluster-data/master-0/wal/instance:
uuid: "850b7724fd5b4d77857edd66e386b90e"
format_stamp: "Formatted at 2026-09-25 22:44:49 on dist-test-slave-5vdr"
server_key: "040384d8dca10f4046a39a49011f9c92"
server_key_iv: "49889fe9af6c2cfb61f3480da9a79be0"
server_key_version: "encryptionkey@0"
I20260925 22:44:49.413888 16591 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260925 22:44:49.417863 16606 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:44:49.419545 16591 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260925 22:44:49.419706 16591 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790376259650642-11032-0/minicluster-data/master-0/data,/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790376259650642-11032-0/minicluster-data/master-0/wal
uuid: "850b7724fd5b4d77857edd66e386b90e"
format_stamp: "Formatted at 2026-09-25 22:44:49 on dist-test-slave-5vdr"
server_key: "040384d8dca10f4046a39a49011f9c92"
server_key_iv: "49889fe9af6c2cfb61f3480da9a79be0"
server_key_version: "encryptionkey@0"
I20260925 22:44:49.419946 16591 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790376259650642-11032-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790376259650642-11032-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790376259650642-11032-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:44:49.432548 16591 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:44:49.436460 16591 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:44:49.436645 16591 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:44:49.444800 16591 rpc_server.cc:307] RPC server started. Bound to: 127.10.198.62:45183
I20260925 22:44:49.444810 16658 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.198.62:45183 every 8 connection(s)
I20260925 22:44:49.445953 16591 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790376259650642-11032-0/minicluster-data/master-0/data/info.pb
I20260925 22:44:49.449339 16659 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:44:49.451924 11032 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrWXm8X/build/debug/bin/kudu as pid 16591
I20260925 22:44:49.452061 11032 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790376259650642-11032-0/minicluster-data/master-0/wal/instance
I20260925 22:44:49.452373 11032 external_mini_cluster.cc:1428] Setting key 2e29aef2f68b256a6c89b0632b35b6b8
I20260925 22:44:49.456688 16659 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 850b7724fd5b4d77857edd66e386b90e: Bootstrap starting.
I20260925 22:44:49.459733 16659 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 850b7724fd5b4d77857edd66e386b90e: Neither blocks nor log segments found. Creating new log.
I20260925 22:44:49.460763 16659 log.cc:824] T 00000000000000000000000000000000 P 850b7724fd5b4d77857edd66e386b90e: Log is configured to *not* fsync() on all Append() calls
Sep 25 22:44:49 dist-test-slave-5vdr krb5kdc[16578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376289, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.198.62@KRBTEST.COM
I20260925 22:44:49.463567 16659 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 850b7724fd5b4d77857edd66e386b90e: No bootstrap required, opened a new log
I20260925 22:44:49.467059 16659 raft_consensus.cc:348] T 00000000000000000000000000000000 P 850b7724fd5b4d77857edd66e386b90e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "850b7724fd5b4d77857edd66e386b90e" member_type: VOTER last_known_addr { host: "127.10.198.62" port: 45183 } }
I20260925 22:44:49.467406 16659 raft_consensus.cc:374] T 00000000000000000000000000000000 P 850b7724fd5b4d77857edd66e386b90e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:44:49.467512 16659 raft_consensus.cc:729] T 00000000000000000000000000000000 P 850b7724fd5b4d77857edd66e386b90e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 850b7724fd5b4d77857edd66e386b90e, State: Initialized, Role: FOLLOWER
I20260925 22:44:49.468195 16659 consensus_queue.cc:260] T 00000000000000000000000000000000 P 850b7724fd5b4d77857edd66e386b90e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "850b7724fd5b4d77857edd66e386b90e" member_type: VOTER last_known_addr { host: "127.10.198.62" port: 45183 } }
I20260925 22:44:49.468426 16659 raft_consensus.cc:388] T 00000000000000000000000000000000 P 850b7724fd5b4d77857edd66e386b90e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 22:44:49.468567 16659 raft_consensus.cc:482] T 00000000000000000000000000000000 P 850b7724fd5b4d77857edd66e386b90e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 22:44:49.468700 16659 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 850b7724fd5b4d77857edd66e386b90e [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:44:49.469799 16662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:44:49.454097 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51180 (local address 127.10.198.62:45183)
0925 22:44:49.454572 (+ 475us) server_negotiation.cc:206] Beginning negotiation
0925 22:44:49.454596 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0925 22:44:49.454639 (+ 43us) server_negotiation.cc:407] Connection header received
0925 22:44:49.455358 (+ 719us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:44:49.455377 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:44:49.455759 (+ 382us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:44:49.456190 (+ 431us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:44:49.457228 (+ 1038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:44:49.458304 (+ 1076us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:44:49.459558 (+ 1254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:44:49.460263 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:44:49.460413 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:44:49.463473 (+ 3060us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:44:49.463479 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:44:49.463496 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:44:49.463530 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:44:49.466146 (+ 2616us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:44:49.466741 (+ 595us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:44:49.466748 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:44:49.466754 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:44:49.466893 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:44:49.467228 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:44:49.467233 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:44:49.467235 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:44:49.467815 (+ 580us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:44:49.468089 (+ 274us) server_negotiation.cc:1035] Waiting for connection context
0925 22:44:49.468551 (+ 462us) server_negotiation.cc:299] Negotiation successful
0925 22:44:49.468942 (+ 391us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":316,"thread_start_us":160,"threads_started":1}
I20260925 22:44:49.469882 16659 raft_consensus.cc:504] T 00000000000000000000000000000000 P 850b7724fd5b4d77857edd66e386b90e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "850b7724fd5b4d77857edd66e386b90e" member_type: VOTER last_known_addr { host: "127.10.198.62" port: 45183 } }
I20260925 22:44:49.470335 16659 leader_election.cc:302] T 00000000000000000000000000000000 P 850b7724fd5b4d77857edd66e386b90e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 850b7724fd5b4d77857edd66e386b90e; no voters:
I20260925 22:44:49.470726 16659 leader_election.cc:288] T 00000000000000000000000000000000 P 850b7724fd5b4d77857edd66e386b90e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 22:44:49.470863 16664 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 850b7724fd5b4d77857edd66e386b90e [term 1 FOLLOWER]: Leader election won for term 1
I20260925 22:44:49.471069 16664 raft_consensus.cc:686] T 00000000000000000000000000000000 P 850b7724fd5b4d77857edd66e386b90e [term 1 LEADER]: Becoming Leader. State: Replica: 850b7724fd5b4d77857edd66e386b90e, State: Running, Role: LEADER
I20260925 22:44:49.471407 16664 consensus_queue.cc:237] T 00000000000000000000000000000000 P 850b7724fd5b4d77857edd66e386b90e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "850b7724fd5b4d77857edd66e386b90e" member_type: VOTER last_known_addr { host: "127.10.198.62" port: 45183 } }
I20260925 22:44:49.472131 16659 sys_catalog.cc:564] T 00000000000000000000000000000000 P 850b7724fd5b4d77857edd66e386b90e [sys.catalog]: configured and running, proceeding with master startup.
I20260925 22:44:49.473485 16665 sys_catalog.cc:455] T 00000000000000000000000000000000 P 850b7724fd5b4d77857edd66e386b90e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "850b7724fd5b4d77857edd66e386b90e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "850b7724fd5b4d77857edd66e386b90e" member_type: VOTER last_known_addr { host: "127.10.198.62" port: 45183 } } }
I20260925 22:44:49.473660 16665 sys_catalog.cc:458] T 00000000000000000000000000000000 P 850b7724fd5b4d77857edd66e386b90e [sys.catalog]: This master's current role is: LEADER
I20260925 22:44:49.474078 16666 sys_catalog.cc:455] T 00000000000000000000000000000000 P 850b7724fd5b4d77857edd66e386b90e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 850b7724fd5b4d77857edd66e386b90e. Latest consensus state: current_term: 1 leader_uuid: "850b7724fd5b4d77857edd66e386b90e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "850b7724fd5b4d77857edd66e386b90e" member_type: VOTER last_known_addr { host: "127.10.198.62" port: 45183 } } }
I20260925 22:44:49.474299 16666 sys_catalog.cc:458] T 00000000000000000000000000000000 P 850b7724fd5b4d77857edd66e386b90e [sys.catalog]: This master's current role is: LEADER
I20260925 22:44:49.474449 16673 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 22:44:49.477849 16673 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 22:44:49.485170 16673 catalog_manager.cc:1347] Generated new cluster ID: 236d7bf986ae46659f7cbe24b7f3e1e9
I20260925 22:44:49.485292 16673 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 22:44:49.498668 16673 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 22:44:49.500398 16673 catalog_manager.cc:1504] Loading token signing keys...
I20260925 22:44:49.523363 16673 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 850b7724fd5b4d77857edd66e386b90e: Generated new TSK 0
I20260925 22:44:49.524405 16673 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.10.198.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.198.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.198.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790376259650642-11032-0/krb5kdc/kudu_127.10.198.1.keytab.
Entry for principal kudu/127.10.198.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790376259650642-11032-0/krb5kdc/kudu_127.10.198.1.keytab.
I20260925 22:44:49.555248 11032 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrWXm8X/build/debug/bin/kudu
/tmp/dist-test-taskrWXm8X/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790376259650642-11032-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790376259650642-11032-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790376259650642-11032-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790376259650642-11032-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.198.1:0
--local_ip_for_outbound_sockets=127.10.198.1
--webserver_interface=127.10.198.1
--webserver_port=0
--tserver_master_addrs=127.10.198.62:45183
--keytab_file=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790376259650642-11032-0/krb5kdc/kudu_127.10.198.1.keytab
--principal=kudu/127.10.198.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.198.20:33605
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790376259650642-11032-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790376259650642-11032-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790376259650642-11032-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:44:49.680135 16685 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:44:49.680466 16685 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:44:49.680514 16685 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:44:49.685052 16685 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:44:49.685137 16685 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:44:49.685246 16685 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.198.1
I20260925 22:44:49.691114 16685 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.198.20:33605
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790376259650642-11032-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790376259650642-11032-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790376259650642-11032-0/krb5kdc/kudu_127.10.198.1.keytab
--principal=kudu/127.10.198.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.198.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790376259650642-11032-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.198.1
--webserver_port=0
--tserver_master_addrs=127.10.198.62:45183
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16685
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.198.1
--log_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790376259650642-11032-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:09:16 UTC on ccycloud.quasar-afmhyw.root.comops.site
build id 4673379
I20260925 22:44:49.692519 16685 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:44:49.693640 16685 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:44:49.704284 16691 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:44:49.704351 16685 server_base.cc:1034] running on GCE node
W20260925 22:44:49.704224 16693 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:44:49.704294 16690 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:44:49.705101 16685 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:44:49.705838 16685 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:44:49.707033 16685 hybrid_clock.cc:648] HybridClock initialized: now 1790376289706997 us; error 56 us; skew 500 ppm
Sep 25 22:44:49 dist-test-slave-5vdr krb5kdc[16578](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376289, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.198.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:44:49.710033 16685 init.cc:377] Logged in from keytab as kudu/127.10.198.1@KRBTEST.COM (short username kudu)
I20260925 22:44:49.711064 16685 webserver.cc:460] Webserver started at http://127.10.198.1:44755/ using document root <none> and password file <none>
I20260925 22:44:49.711690 16685 fs_manager.cc:359] Metadata directory not provided
I20260925 22:44:49.711791 16685 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:44:49.712076 16685 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:44:49.714359 16685 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790376259650642-11032-0/minicluster-data/ts-0/data/instance:
uuid: "ba05599d90ea4d639b4e410f7a27396b"
format_stamp: "Formatted at 2026-09-25 22:44:49 on dist-test-slave-5vdr"
server_key: "074286ecac63c5da65cf67007c60b870"
server_key_iv: "a8499a7b7472d925a69d33cca9f5278f"
server_key_version: "encryptionkey@0"
I20260925 22:44:49.715016 16685 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790376259650642-11032-0/minicluster-data/ts-0/wal/instance:
uuid: "ba05599d90ea4d639b4e410f7a27396b"
format_stamp: "Formatted at 2026-09-25 22:44:49 on dist-test-slave-5vdr"
server_key: "074286ecac63c5da65cf67007c60b870"
server_key_iv: "a8499a7b7472d925a69d33cca9f5278f"
server_key_version: "encryptionkey@0"
I20260925 22:44:49.719276 16685 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260925 22:44:49.722379 16700 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:44:49.723927 16685 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 22:44:49.724102 16685 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790376259650642-11032-0/minicluster-data/ts-0/data,/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790376259650642-11032-0/minicluster-data/ts-0/wal
uuid: "ba05599d90ea4d639b4e410f7a27396b"
format_stamp: "Formatted at 2026-09-25 22:44:49 on dist-test-slave-5vdr"
server_key: "074286ecac63c5da65cf67007c60b870"
server_key_iv: "a8499a7b7472d925a69d33cca9f5278f"
server_key_version: "encryptionkey@0"
I20260925 22:44:49.724246 16685 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790376259650642-11032-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790376259650642-11032-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790376259650642-11032-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:44:49.742710 16685 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:44:49.746294 16685 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:44:49.746533 16685 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:44:49.747175 16685 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 22:44:49.748142 16685 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:44:49.748214 16685 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:44:49.748286 16685 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:44:49.748333 16685 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 22:44:49.757835 16685 rpc_server.cc:307] RPC server started. Bound to: 127.10.198.1:37601
I20260925 22:44:49.757895 16813 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.198.1:37601 every 8 connection(s)
I20260925 22:44:49.758843 16685 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790376259650642-11032-0/minicluster-data/ts-0/data/info.pb
I20260925 22:44:49.763358 11032 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrWXm8X/build/debug/bin/kudu as pid 16685
I20260925 22:44:49.763514 11032 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790376259650642-11032-0/minicluster-data/ts-0/wal/instance
I20260925 22:44:49.763816 11032 external_mini_cluster.cc:1428] Setting key 2d68acc68649eff04fe54d2a564a925a
Sep 25 22:44:49 dist-test-slave-5vdr krb5kdc[16578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376289, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.198.1@KRBTEST.COM for kudu/127.10.198.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.10.198.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.198.2@KRBTEST.COM" created.
I20260925 22:44:49.776684 16662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:44:49.760982 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.198.1:41791 (local address 127.10.198.62:45183)
0925 22:44:49.761168 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0925 22:44:49.761172 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:44:49.762089 (+ 917us) server_negotiation.cc:407] Connection header received
0925 22:44:49.763557 (+ 1468us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:44:49.763561 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:44:49.763627 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:44:49.763759 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:44:49.765519 (+ 1760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:44:49.766279 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:44:49.767935 (+ 1656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:44:49.768216 (+ 281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:44:49.768355 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:44:49.771618 (+ 3263us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:44:49.771623 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:44:49.771625 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:44:49.771657 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:44:49.773574 (+ 1917us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:44:49.774564 (+ 990us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:44:49.774569 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:44:49.774572 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:44:49.774673 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:44:49.775186 (+ 513us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:44:49.775191 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:44:49.775194 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:44:49.775428 (+ 234us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:44:49.775549 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0925 22:44:49.776310 (+ 761us) server_negotiation.cc:299] Negotiation successful
0925 22:44:49.776480 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260925 22:44:49.777701 16816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:44:49.761387 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.198.62:45183 (local address 127.10.198.1:41791)
0925 22:44:49.761898 (+ 511us) negotiation.cc:107] Waiting for socket to connect
0925 22:44:49.761950 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0925 22:44:49.763298 (+ 1348us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:44:49.763989 (+ 691us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:44:49.763999 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:44:49.764565 (+ 566us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:44:49.765315 (+ 750us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:44:49.765341 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:44:49.766464 (+ 1123us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:44:49.766467 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:44:49.767720 (+ 1253us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:44:49.767736 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:44:49.768413 (+ 677us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:44:49.768418 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:44:49.768635 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:44:49.769368 (+ 733us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:44:49.769389 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:44:49.771365 (+ 1976us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:44:49.773881 (+ 2516us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:44:49.773891 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:44:49.773917 (+ 26us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:44:49.774434 (+ 517us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:44:49.774794 (+ 360us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:44:49.774798 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:44:49.774800 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:44:49.775081 (+ 281us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:44:49.775557 (+ 476us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:44:49.775567 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:44:49.776023 (+ 456us) client_negotiation.cc:769] Sending connection context
0925 22:44:49.776256 (+ 233us) client_negotiation.cc:240] Negotiation successful
0925 22:44:49.776713 (+ 457us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":305,"thread_start_us":120,"threads_started":1}
I20260925 22:44:49.779263 16814 heartbeater.cc:347] Connected to a master server at 127.10.198.62:45183
I20260925 22:44:49.779559 16814 heartbeater.cc:464] Registering TS with master...
I20260925 22:44:49.780177 16814 heartbeater.cc:511] Master 127.10.198.62:45183 requested a full tablet report, sending...
I20260925 22:44:49.782174 16623 ts_manager.cc:194] Registered new tserver with Master: ba05599d90ea4d639b4e410f7a27396b (127.10.198.1:37601)
I20260925 22:44:49.783458 16623 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.198.1@KRBTEST.COM'} at 127.10.198.1:41791
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.198.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790376259650642-11032-0/krb5kdc/kudu_127.10.198.2.keytab.
Entry for principal kudu/127.10.198.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790376259650642-11032-0/krb5kdc/kudu_127.10.198.2.keytab.
I20260925 22:44:49.795136 11032 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrWXm8X/build/debug/bin/kudu
/tmp/dist-test-taskrWXm8X/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790376259650642-11032-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790376259650642-11032-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790376259650642-11032-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790376259650642-11032-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.198.2:0
--local_ip_for_outbound_sockets=127.10.198.2
--webserver_interface=127.10.198.2
--webserver_port=0
--tserver_master_addrs=127.10.198.62:45183
--keytab_file=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790376259650642-11032-0/krb5kdc/kudu_127.10.198.2.keytab
--principal=kudu/127.10.198.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.198.20:33605
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790376259650642-11032-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790376259650642-11032-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790376259650642-11032-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:44:49.914543 16819 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:44:49.914923 16819 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:44:49.915009 16819 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:44:49.919603 16819 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:44:49.919734 16819 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:44:49.919833 16819 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.198.2
I20260925 22:44:49.926262 16819 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.198.20:33605
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790376259650642-11032-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790376259650642-11032-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790376259650642-11032-0/krb5kdc/kudu_127.10.198.2.keytab
--principal=kudu/127.10.198.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.198.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790376259650642-11032-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.198.2
--webserver_port=0
--tserver_master_addrs=127.10.198.62:45183
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16819
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.198.2
--log_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790376259650642-11032-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:09:16 UTC on ccycloud.quasar-afmhyw.root.comops.site
build id 4673379
I20260925 22:44:49.927551 16819 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:44:49.928828 16819 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:44:49.939704 16825 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:44:49.939688 16827 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:44:49.939793 16824 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:44:49.940205 16819 server_base.cc:1034] running on GCE node
I20260925 22:44:49.940730 16819 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:44:49.941500 16819 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:44:49.942682 16819 hybrid_clock.cc:648] HybridClock initialized: now 1790376289942642 us; error 59 us; skew 500 ppm
Sep 25 22:44:49 dist-test-slave-5vdr krb5kdc[16578](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376289, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.198.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:44:49.946125 16819 init.cc:377] Logged in from keytab as kudu/127.10.198.2@KRBTEST.COM (short username kudu)
I20260925 22:44:49.947348 16819 webserver.cc:460] Webserver started at http://127.10.198.2:38747/ using document root <none> and password file <none>
I20260925 22:44:49.948110 16819 fs_manager.cc:359] Metadata directory not provided
I20260925 22:44:49.948195 16819 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:44:49.948470 16819 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:44:49.950819 16819 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790376259650642-11032-0/minicluster-data/ts-1/data/instance:
uuid: "81c84a877e3f4e52817be120c0475f69"
format_stamp: "Formatted at 2026-09-25 22:44:49 on dist-test-slave-5vdr"
server_key: "f73ac09ad5da463ecbe9fc7bb5687dad"
server_key_iv: "da9b293a070d23581627d88d5fb08e3f"
server_key_version: "encryptionkey@0"
I20260925 22:44:49.951517 16819 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790376259650642-11032-0/minicluster-data/ts-1/wal/instance:
uuid: "81c84a877e3f4e52817be120c0475f69"
format_stamp: "Formatted at 2026-09-25 22:44:49 on dist-test-slave-5vdr"
server_key: "f73ac09ad5da463ecbe9fc7bb5687dad"
server_key_iv: "da9b293a070d23581627d88d5fb08e3f"
server_key_version: "encryptionkey@0"
I20260925 22:44:49.956118 16819 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.008s sys 0.000s
I20260925 22:44:49.959319 16834 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:44:49.961151 16819 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 22:44:49.961357 16819 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790376259650642-11032-0/minicluster-data/ts-1/data,/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790376259650642-11032-0/minicluster-data/ts-1/wal
uuid: "81c84a877e3f4e52817be120c0475f69"
format_stamp: "Formatted at 2026-09-25 22:44:49 on dist-test-slave-5vdr"
server_key: "f73ac09ad5da463ecbe9fc7bb5687dad"
server_key_iv: "da9b293a070d23581627d88d5fb08e3f"
server_key_version: "encryptionkey@0"
I20260925 22:44:49.961504 16819 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790376259650642-11032-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790376259650642-11032-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790376259650642-11032-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:44:49.991940 16819 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:44:49.995258 16819 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:44:49.995503 16819 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:44:49.996234 16819 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 22:44:49.997151 16819 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:44:49.997203 16819 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:44:49.997278 16819 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:44:49.997313 16819 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 22:44:50.007081 16819 rpc_server.cc:307] RPC server started. Bound to: 127.10.198.2:41931
I20260925 22:44:50.007107 16947 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.198.2:41931 every 8 connection(s)
I20260925 22:44:50.008189 16819 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790376259650642-11032-0/minicluster-data/ts-1/data/info.pb
I20260925 22:44:50.012769 11032 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrWXm8X/build/debug/bin/kudu as pid 16819
I20260925 22:44:50.012877 11032 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790376259650642-11032-0/minicluster-data/ts-1/wal/instance
I20260925 22:44:50.013152 11032 external_mini_cluster.cc:1428] Setting key dd10eab0fff06c14e1c3d6519f425787
Sep 25 22:44:50 dist-test-slave-5vdr krb5kdc[16578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376289, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.198.2@KRBTEST.COM for kudu/127.10.198.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.10.198.3@KRBTEST.COM; defaulting to no policy
I20260925 22:44:50.025653 16662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:44:50.010362 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.198.2:46429 (local address 127.10.198.62:45183)
0925 22:44:50.010540 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0925 22:44:50.010545 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:44:50.011464 (+ 919us) server_negotiation.cc:407] Connection header received
0925 22:44:50.013080 (+ 1616us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:44:50.013084 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:44:50.013155 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:44:50.013256 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:44:50.014662 (+ 1406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:44:50.015185 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:44:50.017185 (+ 2000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:44:50.017596 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:44:50.017754 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:44:50.020865 (+ 3111us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:44:50.020873 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:44:50.020879 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:44:50.020922 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:44:50.022883 (+ 1961us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:44:50.023576 (+ 693us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:44:50.023591 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:44:50.023594 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:44:50.023728 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:44:50.024213 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:44:50.024216 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:44:50.024218 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:44:50.024440 (+ 222us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:44:50.024558 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0925 22:44:50.025264 (+ 706us) server_negotiation.cc:299] Negotiation successful
0925 22:44:50.025435 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260925 22:44:50.026517 16950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:44:50.010721 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.198.62:45183 (local address 127.10.198.2:46429)
0925 22:44:50.011270 (+ 549us) negotiation.cc:107] Waiting for socket to connect
0925 22:44:50.011321 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0925 22:44:50.012717 (+ 1396us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:44:50.013507 (+ 790us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:44:50.013517 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:44:50.013950 (+ 433us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:44:50.014494 (+ 544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:44:50.014510 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:44:50.015343 (+ 833us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:44:50.015347 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:44:50.016954 (+ 1607us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:44:50.016973 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:44:50.017809 (+ 836us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:44:50.017812 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:44:50.017987 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:44:50.018691 (+ 704us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:44:50.018709 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:44:50.020607 (+ 1898us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:44:50.023075 (+ 2468us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:44:50.023084 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:44:50.023107 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:44:50.023447 (+ 340us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:44:50.023880 (+ 433us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:44:50.023883 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:44:50.023884 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:44:50.024082 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:44:50.024556 (+ 474us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:44:50.024565 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:44:50.024970 (+ 405us) client_negotiation.cc:769] Sending connection context
0925 22:44:50.025188 (+ 218us) client_negotiation.cc:240] Negotiation successful
0925 22:44:50.025576 (+ 388us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":323,"thread_start_us":125,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.198.3@KRBTEST.COM" created.
I20260925 22:44:50.027981 16948 heartbeater.cc:347] Connected to a master server at 127.10.198.62:45183
I20260925 22:44:50.028322 16948 heartbeater.cc:464] Registering TS with master...
I20260925 22:44:50.028908 16948 heartbeater.cc:511] Master 127.10.198.62:45183 requested a full tablet report, sending...
I20260925 22:44:50.030442 16623 ts_manager.cc:194] Registered new tserver with Master: 81c84a877e3f4e52817be120c0475f69 (127.10.198.2:41931)
I20260925 22:44:50.031288 16623 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.198.2@KRBTEST.COM'} at 127.10.198.2:46429
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.198.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790376259650642-11032-0/krb5kdc/kudu_127.10.198.3.keytab.
Entry for principal kudu/127.10.198.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790376259650642-11032-0/krb5kdc/kudu_127.10.198.3.keytab.
I20260925 22:44:50.048406 11032 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrWXm8X/build/debug/bin/kudu
/tmp/dist-test-taskrWXm8X/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790376259650642-11032-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790376259650642-11032-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790376259650642-11032-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790376259650642-11032-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.198.3:0
--local_ip_for_outbound_sockets=127.10.198.3
--webserver_interface=127.10.198.3
--webserver_port=0
--tserver_master_addrs=127.10.198.62:45183
--keytab_file=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790376259650642-11032-0/krb5kdc/kudu_127.10.198.3.keytab
--principal=kudu/127.10.198.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.198.20:33605
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790376259650642-11032-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790376259650642-11032-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790376259650642-11032-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:44:50.172670 16953 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:44:50.173192 16953 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:44:50.173296 16953 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:44:50.177909 16953 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:44:50.178004 16953 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:44:50.178071 16953 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.198.3
I20260925 22:44:50.183921 16953 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.198.20:33605
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790376259650642-11032-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790376259650642-11032-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790376259650642-11032-0/krb5kdc/kudu_127.10.198.3.keytab
--principal=kudu/127.10.198.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.198.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790376259650642-11032-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.198.3
--webserver_port=0
--tserver_master_addrs=127.10.198.62:45183
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16953
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.198.3
--log_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790376259650642-11032-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:09:16 UTC on ccycloud.quasar-afmhyw.root.comops.site
build id 4673379
I20260925 22:44:50.185096 16953 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:44:50.186146 16953 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:44:50.197075 16959 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:44:50.197259 16953 server_base.cc:1034] running on GCE node
W20260925 22:44:50.197084 16958 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:44:50.197501 16961 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:44:50.197995 16953 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:44:50.198767 16953 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:44:50.200035 16953 hybrid_clock.cc:648] HybridClock initialized: now 1790376290200008 us; error 48 us; skew 500 ppm
Sep 25 22:44:50 dist-test-slave-5vdr krb5kdc[16578](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376290, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.198.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:44:50.203217 16953 init.cc:377] Logged in from keytab as kudu/127.10.198.3@KRBTEST.COM (short username kudu)
I20260925 22:44:50.204423 16953 webserver.cc:460] Webserver started at http://127.10.198.3:36981/ using document root <none> and password file <none>
I20260925 22:44:50.205065 16953 fs_manager.cc:359] Metadata directory not provided
I20260925 22:44:50.205150 16953 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:44:50.205338 16953 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:44:50.207705 16953 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790376259650642-11032-0/minicluster-data/ts-2/data/instance:
uuid: "da412a0d47e44c78a3d12ba7da290e42"
format_stamp: "Formatted at 2026-09-25 22:44:50 on dist-test-slave-5vdr"
server_key: "6aeb2bc58c3846480085254c73a5dc2d"
server_key_iv: "68f9e170b5f3c9d803fde6daab36c6c5"
server_key_version: "encryptionkey@0"
I20260925 22:44:50.208570 16953 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790376259650642-11032-0/minicluster-data/ts-2/wal/instance:
uuid: "da412a0d47e44c78a3d12ba7da290e42"
format_stamp: "Formatted at 2026-09-25 22:44:50 on dist-test-slave-5vdr"
server_key: "6aeb2bc58c3846480085254c73a5dc2d"
server_key_iv: "68f9e170b5f3c9d803fde6daab36c6c5"
server_key_version: "encryptionkey@0"
I20260925 22:44:50.213439 16953 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260925 22:44:50.216760 16968 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:44:50.218345 16953 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260925 22:44:50.218551 16953 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790376259650642-11032-0/minicluster-data/ts-2/data,/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790376259650642-11032-0/minicluster-data/ts-2/wal
uuid: "da412a0d47e44c78a3d12ba7da290e42"
format_stamp: "Formatted at 2026-09-25 22:44:50 on dist-test-slave-5vdr"
server_key: "6aeb2bc58c3846480085254c73a5dc2d"
server_key_iv: "68f9e170b5f3c9d803fde6daab36c6c5"
server_key_version: "encryptionkey@0"
I20260925 22:44:50.218708 16953 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790376259650642-11032-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790376259650642-11032-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790376259650642-11032-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:44:50.245875 16953 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:44:50.249739 16953 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:44:50.250001 16953 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:44:50.250669 16953 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 22:44:50.251608 16953 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:44:50.251688 16953 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:44:50.251760 16953 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:44:50.251806 16953 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 22:44:50.261909 16953 rpc_server.cc:307] RPC server started. Bound to: 127.10.198.3:40353
I20260925 22:44:50.261933 17081 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.198.3:40353 every 8 connection(s)
I20260925 22:44:50.262948 16953 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790376259650642-11032-0/minicluster-data/ts-2/data/info.pb
I20260925 22:44:50.266422 11032 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrWXm8X/build/debug/bin/kudu as pid 16953
I20260925 22:44:50.266559 11032 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790376259650642-11032-0/minicluster-data/ts-2/wal/instance
I20260925 22:44:50.266878 11032 external_mini_cluster.cc:1428] Setting key 40c101efa6126c622aaf0f66598ff607
Sep 25 22:44:50 dist-test-slave-5vdr krb5kdc[16578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376290, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.198.3@KRBTEST.COM for kudu/127.10.198.62@KRBTEST.COM
I20260925 22:44:50.280445 16662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:44:50.265098 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.198.3:36937 (local address 127.10.198.62:45183)
0925 22:44:50.265261 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0925 22:44:50.265265 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:44:50.266430 (+ 1165us) server_negotiation.cc:407] Connection header received
0925 22:44:50.268246 (+ 1816us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:44:50.268249 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:44:50.268304 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:44:50.268391 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:44:50.269951 (+ 1560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:44:50.270488 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:44:50.272099 (+ 1611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:44:50.272510 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:44:50.272634 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:44:50.275975 (+ 3341us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:44:50.275979 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:44:50.275981 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:44:50.276013 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:44:50.277732 (+ 1719us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:44:50.278451 (+ 719us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:44:50.278456 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:44:50.278458 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:44:50.278569 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:44:50.279015 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:44:50.279019 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:44:50.279021 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:44:50.279251 (+ 230us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:44:50.279373 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0925 22:44:50.280074 (+ 701us) server_negotiation.cc:299] Negotiation successful
0925 22:44:50.280248 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260925 22:44:50.281512 17084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:44:50.265556 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.198.62:45183 (local address 127.10.198.3:36937)
0925 22:44:50.266210 (+ 654us) negotiation.cc:107] Waiting for socket to connect
0925 22:44:50.266284 (+ 74us) client_negotiation.cc:174] Beginning negotiation
0925 22:44:50.268015 (+ 1731us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:44:50.268617 (+ 602us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:44:50.268628 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:44:50.269102 (+ 474us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:44:50.269794 (+ 692us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:44:50.269813 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:44:50.270620 (+ 807us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:44:50.270623 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:44:50.271644 (+ 1021us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:44:50.271655 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:44:50.272723 (+ 1068us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:44:50.272727 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:44:50.272931 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:44:50.273760 (+ 829us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:44:50.273781 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:44:50.275660 (+ 1879us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:44:50.277917 (+ 2257us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:44:50.277925 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:44:50.277948 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:44:50.278310 (+ 362us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:44:50.278695 (+ 385us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:44:50.278699 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:44:50.278701 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:44:50.278888 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:44:50.279387 (+ 499us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:44:50.279397 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:44:50.279713 (+ 316us) client_negotiation.cc:769] Sending connection context
0925 22:44:50.280013 (+ 300us) client_negotiation.cc:240] Negotiation successful
0925 22:44:50.280408 (+ 395us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":372,"thread_start_us":127,"threads_started":1}
I20260925 22:44:50.282737 17082 heartbeater.cc:347] Connected to a master server at 127.10.198.62:45183
I20260925 22:44:50.283051 17082 heartbeater.cc:464] Registering TS with master...
I20260925 22:44:50.283610 17082 heartbeater.cc:511] Master 127.10.198.62:45183 requested a full tablet report, sending...
I20260925 22:44:50.284876 16623 ts_manager.cc:194] Registered new tserver with Master: da412a0d47e44c78a3d12ba7da290e42 (127.10.198.3:40353)
I20260925 22:44:50.285478 16623 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.198.3@KRBTEST.COM'} at 127.10.198.3:36937
I20260925 22:44:50.294250 11032 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790376259650642-11032-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790376259650642-11032-0/krb5kdc/test-admin.keytab.
Sep 25 22:44:50 dist-test-slave-5vdr krb5kdc[16578](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376290, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:44:50.312557 11032 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260925 22:44:50.315555 17089 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:44:50.314125 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45086 (local address 127.10.198.1:37601)
0925 22:44:50.314417 (+ 292us) server_negotiation.cc:206] Beginning negotiation
0925 22:44:50.314425 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 22:44:50.314455 (+ 30us) server_negotiation.cc:407] Connection header received
0925 22:44:50.314519 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:44:50.314527 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:44:50.314733 (+ 206us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:44:50.314928 (+ 195us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:44:50.315410 (+ 482us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45086: BlockingRecv error: recv got EOF from 127.0.0.1:45086 (error 108)
Metrics: {"server-negotiator.queue_time_us":183,"thread_start_us":74,"threads_started":1}
W20260925 22:44:50.316097 11032 init.cc:244] Error while opening credential cache '/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790376259650642-11032-0/krb5kdc/krb5cc' of type FILE: Runtime error: End of credential cache reached (filename: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790376259650642-11032-0/krb5kdc/krb5cc)
Sep 25 22:44:50 dist-test-slave-5vdr krb5kdc[16578](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376290, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:44:50.317556 11032 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 25 22:44:50 dist-test-slave-5vdr krb5kdc[16578](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376290, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:44:50.318733 11032 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 25 22:44:50 dist-test-slave-5vdr krb5kdc[16578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376290, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.198.1@KRBTEST.COM
I20260925 22:44:50.330039 17089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:44:50.319994 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45094 (local address 127.10.198.1:37601)
0925 22:44:50.320144 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0925 22:44:50.320148 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:44:50.320240 (+ 92us) server_negotiation.cc:407] Connection header received
0925 22:44:50.320342 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:44:50.320345 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:44:50.320403 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:44:50.320482 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:44:50.321055 (+ 573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:44:50.321855 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:44:50.322853 (+ 998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:44:50.323379 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:44:50.323529 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:44:50.326361 (+ 2832us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:44:50.326373 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:44:50.326388 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:44:50.326422 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:44:50.328169 (+ 1747us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:44:50.328668 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:44:50.328673 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:44:50.328674 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:44:50.328783 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:44:50.329061 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:44:50.329064 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:44:50.329065 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:44:50.329408 (+ 343us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:44:50.329578 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0925 22:44:50.329778 (+ 200us) server_negotiation.cc:299] Negotiation successful
0925 22:44:50.329913 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
W20260925 22:44:50.334450 17089 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:44:50.333260 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45110 (local address 127.10.198.1:37601)
0925 22:44:50.333390 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0925 22:44:50.333394 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:44:50.333552 (+ 158us) server_negotiation.cc:407] Connection header received
0925 22:44:50.333667 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:44:50.333669 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:44:50.333709 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:44:50.333792 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:44:50.334313 (+ 521us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45110: BlockingRecv error: recv got EOF from 127.0.0.1:45110 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
W20260925 22:44:50.334542 11032 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790376259650642-11032-0/krb5kdc/krb5cc' of type FILE
Sep 25 22:44:50 dist-test-slave-5vdr krb5kdc[16578](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376290, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:44:50.335664 11032 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 25 22:44:50 dist-test-slave-5vdr krb5kdc[16578](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376290, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:44:50.336709 11032 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 25 22:44:50 dist-test-slave-5vdr krb5kdc[16578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376290, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.198.1@KRBTEST.COM
I20260925 22:44:50.346144 17089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:44:50.337743 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45116 (local address 127.10.198.1:37601)
0925 22:44:50.337893 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0925 22:44:50.337897 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:44:50.337911 (+ 14us) server_negotiation.cc:407] Connection header received
0925 22:44:50.338025 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:44:50.338028 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:44:50.338067 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:44:50.338135 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:44:50.338635 (+ 500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:44:50.339140 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:44:50.340054 (+ 914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:44:50.340279 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:44:50.340393 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:44:50.342929 (+ 2536us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:44:50.342932 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:44:50.342934 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:44:50.342961 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:44:50.344375 (+ 1414us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:44:50.344840 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:44:50.344843 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:44:50.344845 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:44:50.344936 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:44:50.345267 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:44:50.345270 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:44:50.345271 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:44:50.345463 (+ 192us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:44:50.345586 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0925 22:44:50.345798 (+ 212us) server_negotiation.cc:299] Negotiation successful
0925 22:44:50.345902 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
W20260925 22:44:50.349584 17089 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:44:50.348370 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45118 (local address 127.10.198.1:37601)
0925 22:44:50.348520 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0925 22:44:50.348524 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:44:50.348646 (+ 122us) server_negotiation.cc:407] Connection header received
0925 22:44:50.348804 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:44:50.348808 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:44:50.348857 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:44:50.348962 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:44:50.349444 (+ 482us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45118: BlockingRecv error: recv got EOF from 127.0.0.1:45118 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
W20260925 22:44:50.349828 11032 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790376259650642-11032-0/krb5kdc/krb5cc' of type FILE
Sep 25 22:44:50 dist-test-slave-5vdr krb5kdc[16578](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376290, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:44:50.350942 11032 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 25 22:44:50 dist-test-slave-5vdr krb5kdc[16578](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376290, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:44:50.351936 11032 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 25 22:44:50 dist-test-slave-5vdr krb5kdc[16578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376290, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.198.1@KRBTEST.COM
I20260925 22:44:50.361493 17089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:44:50.352984 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45120 (local address 127.10.198.1:37601)
0925 22:44:50.353130 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0925 22:44:50.353140 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 22:44:50.353161 (+ 21us) server_negotiation.cc:407] Connection header received
0925 22:44:50.353291 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:44:50.353294 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:44:50.353336 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:44:50.353408 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:44:50.353955 (+ 547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:44:50.354469 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:44:50.355290 (+ 821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:44:50.355511 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:44:50.355601 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:44:50.358112 (+ 2511us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:44:50.358116 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:44:50.358118 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:44:50.358151 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:44:50.359623 (+ 1472us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:44:50.360146 (+ 523us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:44:50.360149 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:44:50.360151 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:44:50.360237 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:44:50.360540 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:44:50.360542 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:44:50.360543 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:44:50.360740 (+ 197us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:44:50.360899 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0925 22:44:50.361175 (+ 276us) server_negotiation.cc:299] Negotiation successful
0925 22:44:50.361287 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260925 22:44:50.362668 11032 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrWXm8X/build/debug/bin/kudu with pid 16685
I20260925 22:44:50.368994 11032 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrWXm8X/build/debug/bin/kudu with pid 16819
I20260925 22:44:50.375280 11032 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrWXm8X/build/debug/bin/kudu with pid 16953
I20260925 22:44:50.381290 11032 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrWXm8X/build/debug/bin/kudu with pid 16591
2026-09-25T22:44:50Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (2336 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376259650642-11032-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 22:44:50 dist-test-slave-5vdr krb5kdc[17102](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 22:44:50 dist-test-slave-5vdr krb5kdc[17102](info): set up 2 sockets
Sep 25 22:44:50 dist-test-slave-5vdr krb5kdc[17102](info): commencing operation
krb5kdc: starting...
W20260925 22:44:51.436648 11032 mini_kdc.cc:121] Time spent starting KDC: real 1.032s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376259650642-11032-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376259650642-11032-0/krb5kdc/test-user.keytab.
Sep 25 22:44:51 dist-test-slave-5vdr krb5kdc[17102](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376291, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T22:44:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T22:44:51Z Disabled control of system clock
WARNING: no policy specified for oryx/127.10.198.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.10.198.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.10.198.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376259650642-11032-0/krb5kdc/oryx_127.10.198.62.keytab.
Entry for principal oryx/127.10.198.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376259650642-11032-0/krb5kdc/oryx_127.10.198.62.keytab.
I20260925 22:44:51.569235 11032 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrWXm8X/build/debug/bin/kudu
/tmp/dist-test-taskrWXm8X/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376259650642-11032-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376259650642-11032-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376259650642-11032-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376259650642-11032-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.198.62:41969
--webserver_interface=127.10.198.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376259650642-11032-0/krb5kdc/oryx_127.10.198.62.keytab
--principal=oryx/127.10.198.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.198.20:44855
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.198.62:41969
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376259650642-11032-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376259650642-11032-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376259650642-11032-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:44:51.690629 17115 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:44:51.690960 17115 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:44:51.691005 17115 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:44:51.695564 17115 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 22:44:51.695642 17115 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:44:51.695662 17115 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:44:51.695680 17115 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 22:44:51.695696 17115 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 22:44:51.702209 17115 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.198.20:44855
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376259650642-11032-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376259650642-11032-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.198.62:41969
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376259650642-11032-0/krb5kdc/oryx_127.10.198.62.keytab
--principal=oryx/127.10.198.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.198.62:41969
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376259650642-11032-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.198.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17115
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376259650642-11032-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:09:16 UTC on ccycloud.quasar-afmhyw.root.comops.site
build id 4673379
I20260925 22:44:51.703446 17115 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:44:51.704631 17115 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:44:51.713954 17121 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:44:51.713954 17123 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:44:51.713969 17120 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:44:51.714002 17115 server_base.cc:1034] running on GCE node
I20260925 22:44:51.714845 17115 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:44:51.716181 17115 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:44:51.717377 17115 hybrid_clock.cc:648] HybridClock initialized: now 1790376291717344 us; error 52 us; skew 500 ppm
Sep 25 22:44:51 dist-test-slave-5vdr krb5kdc[17102](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376291, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.198.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:44:51.720647 17115 init.cc:377] Logged in from keytab as oryx/127.10.198.62@KRBTEST.COM (short username oryx)
I20260925 22:44:51.721727 17115 webserver.cc:460] Webserver started at http://127.10.198.62:33553/ using document root <none> and password file <none>
I20260925 22:44:51.722375 17115 fs_manager.cc:359] Metadata directory not provided
I20260925 22:44:51.722484 17115 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:44:51.722745 17115 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:44:51.725135 17115 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376259650642-11032-0/minicluster-data/master-0/data/instance:
uuid: "36158ad8c29246c2ab5424b4baf3038d"
format_stamp: "Formatted at 2026-09-25 22:44:51 on dist-test-slave-5vdr"
server_key: "01f2739f5f967adae73d889dfc7b32af"
server_key_iv: "3cb386b2cb016532313e6ed2db5ad509"
server_key_version: "encryptionkey@0"
I20260925 22:44:51.725819 17115 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376259650642-11032-0/minicluster-data/master-0/wal/instance:
uuid: "36158ad8c29246c2ab5424b4baf3038d"
format_stamp: "Formatted at 2026-09-25 22:44:51 on dist-test-slave-5vdr"
server_key: "01f2739f5f967adae73d889dfc7b32af"
server_key_iv: "3cb386b2cb016532313e6ed2db5ad509"
server_key_version: "encryptionkey@0"
I20260925 22:44:51.730264 17115 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260925 22:44:51.733230 17130 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:44:51.734784 17115 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260925 22:44:51.734984 17115 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376259650642-11032-0/minicluster-data/master-0/data,/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376259650642-11032-0/minicluster-data/master-0/wal
uuid: "36158ad8c29246c2ab5424b4baf3038d"
format_stamp: "Formatted at 2026-09-25 22:44:51 on dist-test-slave-5vdr"
server_key: "01f2739f5f967adae73d889dfc7b32af"
server_key_iv: "3cb386b2cb016532313e6ed2db5ad509"
server_key_version: "encryptionkey@0"
I20260925 22:44:51.735138 17115 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376259650642-11032-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376259650642-11032-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376259650642-11032-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:44:51.763536 17115 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:44:51.767241 17115 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:44:51.767484 17115 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:44:51.776285 17115 rpc_server.cc:307] RPC server started. Bound to: 127.10.198.62:41969
I20260925 22:44:51.776306 17182 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.198.62:41969 every 8 connection(s)
I20260925 22:44:51.777401 17115 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376259650642-11032-0/minicluster-data/master-0/data/info.pb
I20260925 22:44:51.780725 17183 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:44:51.787124 11032 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrWXm8X/build/debug/bin/kudu as pid 17115
I20260925 22:44:51.787114 17183 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 36158ad8c29246c2ab5424b4baf3038d: Bootstrap starting.
I20260925 22:44:51.787333 11032 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376259650642-11032-0/minicluster-data/master-0/wal/instance
I20260925 22:44:51.787636 11032 external_mini_cluster.cc:1428] Setting key 2bd859b575bc50f0cd17a2b7d6511885
I20260925 22:44:51.790190 17183 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 36158ad8c29246c2ab5424b4baf3038d: Neither blocks nor log segments found. Creating new log.
I20260925 22:44:51.791023 17183 log.cc:824] T 00000000000000000000000000000000 P 36158ad8c29246c2ab5424b4baf3038d: Log is configured to *not* fsync() on all Append() calls
I20260925 22:44:51.793215 17183 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 36158ad8c29246c2ab5424b4baf3038d: No bootstrap required, opened a new log
Sep 25 22:44:51 dist-test-slave-5vdr krb5kdc[17102](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376291, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.10.198.62@KRBTEST.COM
I20260925 22:44:51.798722 17183 raft_consensus.cc:348] T 00000000000000000000000000000000 P 36158ad8c29246c2ab5424b4baf3038d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36158ad8c29246c2ab5424b4baf3038d" member_type: VOTER last_known_addr { host: "127.10.198.62" port: 41969 } }
I20260925 22:44:51.799124 17183 raft_consensus.cc:374] T 00000000000000000000000000000000 P 36158ad8c29246c2ab5424b4baf3038d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:44:51.799223 17183 raft_consensus.cc:729] T 00000000000000000000000000000000 P 36158ad8c29246c2ab5424b4baf3038d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 36158ad8c29246c2ab5424b4baf3038d, State: Initialized, Role: FOLLOWER
I20260925 22:44:51.799902 17183 consensus_queue.cc:260] T 00000000000000000000000000000000 P 36158ad8c29246c2ab5424b4baf3038d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36158ad8c29246c2ab5424b4baf3038d" member_type: VOTER last_known_addr { host: "127.10.198.62" port: 41969 } }
I20260925 22:44:51.800133 17183 raft_consensus.cc:388] T 00000000000000000000000000000000 P 36158ad8c29246c2ab5424b4baf3038d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 22:44:51.800235 17183 raft_consensus.cc:482] T 00000000000000000000000000000000 P 36158ad8c29246c2ab5424b4baf3038d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 22:44:51.800359 17183 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 36158ad8c29246c2ab5424b4baf3038d [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:44:51.801824 17183 raft_consensus.cc:504] T 00000000000000000000000000000000 P 36158ad8c29246c2ab5424b4baf3038d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36158ad8c29246c2ab5424b4baf3038d" member_type: VOTER last_known_addr { host: "127.10.198.62" port: 41969 } }
I20260925 22:44:51.802313 17183 leader_election.cc:302] T 00000000000000000000000000000000 P 36158ad8c29246c2ab5424b4baf3038d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 36158ad8c29246c2ab5424b4baf3038d; no voters:
I20260925 22:44:51.802726 17183 leader_election.cc:288] T 00000000000000000000000000000000 P 36158ad8c29246c2ab5424b4baf3038d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 22:44:51.802866 17188 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 36158ad8c29246c2ab5424b4baf3038d [term 1 FOLLOWER]: Leader election won for term 1
I20260925 22:44:51.803093 17188 raft_consensus.cc:686] T 00000000000000000000000000000000 P 36158ad8c29246c2ab5424b4baf3038d [term 1 LEADER]: Becoming Leader. State: Replica: 36158ad8c29246c2ab5424b4baf3038d, State: Running, Role: LEADER
I20260925 22:44:51.803542 17188 consensus_queue.cc:237] T 00000000000000000000000000000000 P 36158ad8c29246c2ab5424b4baf3038d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36158ad8c29246c2ab5424b4baf3038d" member_type: VOTER last_known_addr { host: "127.10.198.62" port: 41969 } }
I20260925 22:44:51.804265 17183 sys_catalog.cc:564] T 00000000000000000000000000000000 P 36158ad8c29246c2ab5424b4baf3038d [sys.catalog]: configured and running, proceeding with master startup.
I20260925 22:44:51.805584 17186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:44:51.789139 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42054 (local address 127.10.198.62:41969)
0925 22:44:51.789646 (+ 507us) server_negotiation.cc:206] Beginning negotiation
0925 22:44:51.789667 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0925 22:44:51.789706 (+ 39us) server_negotiation.cc:407] Connection header received
0925 22:44:51.790515 (+ 809us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:44:51.790537 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:44:51.790925 (+ 388us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:44:51.791320 (+ 395us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:44:51.792244 (+ 924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:44:51.793255 (+ 1011us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:44:51.794471 (+ 1216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:44:51.794984 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:44:51.795122 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:44:51.798731 (+ 3609us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:44:51.798739 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:44:51.798775 (+ 36us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:44:51.798823 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:44:51.801327 (+ 2504us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:44:51.802013 (+ 686us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:44:51.802019 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:44:51.802026 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:44:51.802177 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:44:51.802525 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:44:51.802528 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:44:51.802529 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:44:51.803175 (+ 646us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:44:51.803439 (+ 264us) server_negotiation.cc:1035] Waiting for connection context
0925 22:44:51.804065 (+ 626us) server_negotiation.cc:299] Negotiation successful
0925 22:44:51.804429 (+ 364us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":347,"thread_start_us":172,"threads_started":1}
I20260925 22:44:51.807632 17190 sys_catalog.cc:455] T 00000000000000000000000000000000 P 36158ad8c29246c2ab5424b4baf3038d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 36158ad8c29246c2ab5424b4baf3038d. Latest consensus state: current_term: 1 leader_uuid: "36158ad8c29246c2ab5424b4baf3038d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36158ad8c29246c2ab5424b4baf3038d" member_type: VOTER last_known_addr { host: "127.10.198.62" port: 41969 } } }
I20260925 22:44:51.807814 17190 sys_catalog.cc:458] T 00000000000000000000000000000000 P 36158ad8c29246c2ab5424b4baf3038d [sys.catalog]: This master's current role is: LEADER
I20260925 22:44:51.807623 17189 sys_catalog.cc:455] T 00000000000000000000000000000000 P 36158ad8c29246c2ab5424b4baf3038d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "36158ad8c29246c2ab5424b4baf3038d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36158ad8c29246c2ab5424b4baf3038d" member_type: VOTER last_known_addr { host: "127.10.198.62" port: 41969 } } }
I20260925 22:44:51.808053 17189 sys_catalog.cc:458] T 00000000000000000000000000000000 P 36158ad8c29246c2ab5424b4baf3038d [sys.catalog]: This master's current role is: LEADER
W20260925 22:44:51.810571 17202 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 36158ad8c29246c2ab5424b4baf3038d: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 22:44:51.810673 17202 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 22:44:51.810773 17204 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 22:44:51.811609 17204 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 22:44:51.818590 17204 catalog_manager.cc:1347] Generated new cluster ID: dec13579c22b435db994a21e4ce3e494
I20260925 22:44:51.818688 17204 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 22:44:51.848037 17204 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 22:44:51.849474 17204 catalog_manager.cc:1504] Loading token signing keys...
I20260925 22:44:51.860519 17204 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 36158ad8c29246c2ab5424b4baf3038d: Generated new TSK 0
I20260925 22:44:51.861521 17204 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.10.198.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.10.198.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.10.198.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376259650642-11032-0/krb5kdc/oryx_127.10.198.1.keytab.
Entry for principal oryx/127.10.198.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376259650642-11032-0/krb5kdc/oryx_127.10.198.1.keytab.
I20260925 22:44:51.899376 11032 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrWXm8X/build/debug/bin/kudu
/tmp/dist-test-taskrWXm8X/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376259650642-11032-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376259650642-11032-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376259650642-11032-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376259650642-11032-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.198.1:0
--local_ip_for_outbound_sockets=127.10.198.1
--webserver_interface=127.10.198.1
--webserver_port=0
--tserver_master_addrs=127.10.198.62:41969
--keytab_file=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376259650642-11032-0/krb5kdc/oryx_127.10.198.1.keytab
--principal=oryx/127.10.198.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.198.20:44855
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376259650642-11032-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376259650642-11032-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376259650642-11032-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:44:52.026328 17209 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:44:52.026688 17209 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:44:52.026768 17209 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:44:52.031189 17209 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:44:52.031337 17209 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:44:52.031453 17209 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.198.1
I20260925 22:44:52.037506 17209 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.198.20:44855
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376259650642-11032-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376259650642-11032-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376259650642-11032-0/krb5kdc/oryx_127.10.198.1.keytab
--principal=oryx/127.10.198.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.198.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376259650642-11032-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.198.1
--webserver_port=0
--tserver_master_addrs=127.10.198.62:41969
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17209
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.198.1
--log_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376259650642-11032-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:09:16 UTC on ccycloud.quasar-afmhyw.root.comops.site
build id 4673379
I20260925 22:44:52.038823 17209 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:44:52.040028 17209 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:44:52.050760 17214 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:44:52.050760 17215 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:44:52.050887 17217 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:44:52.051234 17209 server_base.cc:1034] running on GCE node
I20260925 22:44:52.051755 17209 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:44:52.052611 17209 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:44:52.053812 17209 hybrid_clock.cc:648] HybridClock initialized: now 1790376292053797 us; error 36 us; skew 500 ppm
Sep 25 22:44:52 dist-test-slave-5vdr krb5kdc[17102](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376292, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.198.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:44:52.057466 17209 init.cc:377] Logged in from keytab as oryx/127.10.198.1@KRBTEST.COM (short username oryx)
I20260925 22:44:52.058568 17209 webserver.cc:460] Webserver started at http://127.10.198.1:35903/ using document root <none> and password file <none>
I20260925 22:44:52.059245 17209 fs_manager.cc:359] Metadata directory not provided
I20260925 22:44:52.059360 17209 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:44:52.059617 17209 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:44:52.062253 17209 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376259650642-11032-0/minicluster-data/ts-0/data/instance:
uuid: "dcfd1608a17742d9a03fa549b12bb0c8"
format_stamp: "Formatted at 2026-09-25 22:44:52 on dist-test-slave-5vdr"
server_key: "945983d5c65a0dacba5b4f65b51478ca"
server_key_iv: "f34c5c64750b11fdda091a0914bba73f"
server_key_version: "encryptionkey@0"
I20260925 22:44:52.063030 17209 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376259650642-11032-0/minicluster-data/ts-0/wal/instance:
uuid: "dcfd1608a17742d9a03fa549b12bb0c8"
format_stamp: "Formatted at 2026-09-25 22:44:52 on dist-test-slave-5vdr"
server_key: "945983d5c65a0dacba5b4f65b51478ca"
server_key_iv: "f34c5c64750b11fdda091a0914bba73f"
server_key_version: "encryptionkey@0"
I20260925 22:44:52.067972 17209 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.008s sys 0.000s
I20260925 22:44:52.071415 17224 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:44:52.073138 17209 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 22:44:52.073359 17209 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376259650642-11032-0/minicluster-data/ts-0/data,/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376259650642-11032-0/minicluster-data/ts-0/wal
uuid: "dcfd1608a17742d9a03fa549b12bb0c8"
format_stamp: "Formatted at 2026-09-25 22:44:52 on dist-test-slave-5vdr"
server_key: "945983d5c65a0dacba5b4f65b51478ca"
server_key_iv: "f34c5c64750b11fdda091a0914bba73f"
server_key_version: "encryptionkey@0"
I20260925 22:44:52.073518 17209 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376259650642-11032-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376259650642-11032-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376259650642-11032-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:44:52.098956 17209 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:44:52.102897 17209 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:44:52.103144 17209 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:44:52.103915 17209 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 22:44:52.105201 17209 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:44:52.105276 17209 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:44:52.105342 17209 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:44:52.105376 17209 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 22:44:52.116005 17209 rpc_server.cc:307] RPC server started. Bound to: 127.10.198.1:41135
I20260925 22:44:52.116073 17337 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.198.1:41135 every 8 connection(s)
I20260925 22:44:52.117157 17209 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376259650642-11032-0/minicluster-data/ts-0/data/info.pb
I20260925 22:44:52.117727 11032 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrWXm8X/build/debug/bin/kudu as pid 17209
I20260925 22:44:52.117873 11032 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376259650642-11032-0/minicluster-data/ts-0/wal/instance
I20260925 22:44:52.118228 11032 external_mini_cluster.cc:1428] Setting key be73a9ffec7027869071654f9f3e52e0
WARNING: no policy specified for oryx/127.10.198.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.10.198.2@KRBTEST.COM" created.
Sep 25 22:44:52 dist-test-slave-5vdr krb5kdc[17102](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376292, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.198.1@KRBTEST.COM for oryx/127.10.198.62@KRBTEST.COM
I20260925 22:44:52.138202 17186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:44:52.119499 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.198.1:53687 (local address 127.10.198.62:41969)
0925 22:44:52.119654 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0925 22:44:52.119659 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:44:52.120904 (+ 1245us) server_negotiation.cc:407] Connection header received
0925 22:44:52.123019 (+ 2115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:44:52.123022 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:44:52.123095 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:44:52.123184 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:44:52.125328 (+ 2144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:44:52.126344 (+ 1016us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:44:52.127800 (+ 1456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:44:52.128303 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:44:52.128460 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:44:52.133787 (+ 5327us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:44:52.133790 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:44:52.133793 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:44:52.133825 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:44:52.135549 (+ 1724us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:44:52.136369 (+ 820us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:44:52.136372 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:44:52.136374 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:44:52.136471 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:44:52.136899 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:44:52.136903 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:44:52.136905 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:44:52.137131 (+ 226us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:44:52.137255 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0925 22:44:52.137863 (+ 608us) server_negotiation.cc:299] Negotiation successful
0925 22:44:52.138021 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260925 22:44:52.139318 17340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:44:52.120142 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.198.62:41969 (local address 127.10.198.1:53687)
0925 22:44:52.120690 (+ 548us) negotiation.cc:107] Waiting for socket to connect
0925 22:44:52.120758 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0925 22:44:52.122692 (+ 1934us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:44:52.123441 (+ 749us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:44:52.123456 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:44:52.124124 (+ 668us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:44:52.125022 (+ 898us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:44:52.125050 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:44:52.126596 (+ 1546us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:44:52.126600 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:44:52.127601 (+ 1001us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:44:52.127615 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:44:52.128551 (+ 936us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:44:52.128570 (+ 19us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:44:52.128783 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:44:52.129588 (+ 805us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:44:52.129618 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:44:52.133501 (+ 3883us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:44:52.135739 (+ 2238us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:44:52.135748 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:44:52.135771 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:44:52.136227 (+ 456us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:44:52.136584 (+ 357us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:44:52.136587 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:44:52.136590 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:44:52.136785 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:44:52.137264 (+ 479us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:44:52.137275 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:44:52.137584 (+ 309us) client_negotiation.cc:769] Sending connection context
0925 22:44:52.137803 (+ 219us) client_negotiation.cc:240] Negotiation successful
0925 22:44:52.138192 (+ 389us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":315,"thread_start_us":161,"threads_started":1}
I20260925 22:44:52.141068 17338 heartbeater.cc:347] Connected to a master server at 127.10.198.62:41969
I20260925 22:44:52.141366 17338 heartbeater.cc:464] Registering TS with master...
I20260925 22:44:52.142002 17338 heartbeater.cc:511] Master 127.10.198.62:41969 requested a full tablet report, sending...
I20260925 22:44:52.144205 17147 ts_manager.cc:194] Registered new tserver with Master: dcfd1608a17742d9a03fa549b12bb0c8 (127.10.198.1:41135)
I20260925 22:44:52.145839 17147 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.10.198.1@KRBTEST.COM'} at 127.10.198.1:53687
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.10.198.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376259650642-11032-0/krb5kdc/oryx_127.10.198.2.keytab.
Entry for principal oryx/127.10.198.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376259650642-11032-0/krb5kdc/oryx_127.10.198.2.keytab.
I20260925 22:44:52.151383 11032 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrWXm8X/build/debug/bin/kudu
/tmp/dist-test-taskrWXm8X/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376259650642-11032-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376259650642-11032-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376259650642-11032-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376259650642-11032-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.198.2:0
--local_ip_for_outbound_sockets=127.10.198.2
--webserver_interface=127.10.198.2
--webserver_port=0
--tserver_master_addrs=127.10.198.62:41969
--keytab_file=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376259650642-11032-0/krb5kdc/oryx_127.10.198.2.keytab
--principal=oryx/127.10.198.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.198.20:44855
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376259650642-11032-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376259650642-11032-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376259650642-11032-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:44:52.270586 17343 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:44:52.270903 17343 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:44:52.270951 17343 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:44:52.275799 17343 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:44:52.275923 17343 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:44:52.275990 17343 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.198.2
I20260925 22:44:52.281991 17343 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.198.20:44855
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376259650642-11032-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376259650642-11032-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376259650642-11032-0/krb5kdc/oryx_127.10.198.2.keytab
--principal=oryx/127.10.198.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.198.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376259650642-11032-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.198.2
--webserver_port=0
--tserver_master_addrs=127.10.198.62:41969
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17343
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.198.2
--log_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376259650642-11032-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:09:16 UTC on ccycloud.quasar-afmhyw.root.comops.site
build id 4673379
I20260925 22:44:52.283144 17343 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:44:52.284242 17343 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:44:52.294541 17348 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:44:52.294541 17349 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:44:52.294545 17351 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:44:52.294941 17343 server_base.cc:1034] running on GCE node
I20260925 22:44:52.295436 17343 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:44:52.296281 17343 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:44:52.297519 17343 hybrid_clock.cc:648] HybridClock initialized: now 1790376292297489 us; error 44 us; skew 500 ppm
Sep 25 22:44:52 dist-test-slave-5vdr krb5kdc[17102](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376292, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.198.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:44:52.300838 17343 init.cc:377] Logged in from keytab as oryx/127.10.198.2@KRBTEST.COM (short username oryx)
I20260925 22:44:52.302402 17343 webserver.cc:460] Webserver started at http://127.10.198.2:37641/ using document root <none> and password file <none>
I20260925 22:44:52.303138 17343 fs_manager.cc:359] Metadata directory not provided
I20260925 22:44:52.303263 17343 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:44:52.303570 17343 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:44:52.306494 17343 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376259650642-11032-0/minicluster-data/ts-1/data/instance:
uuid: "f5cc83f491e5409895f3eaf63bdc05c6"
format_stamp: "Formatted at 2026-09-25 22:44:52 on dist-test-slave-5vdr"
server_key: "650c629a002fafdce07b55584ae7301c"
server_key_iv: "a7503e5c9704624c1a465fd5d79e4478"
server_key_version: "encryptionkey@0"
I20260925 22:44:52.307211 17343 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376259650642-11032-0/minicluster-data/ts-1/wal/instance:
uuid: "f5cc83f491e5409895f3eaf63bdc05c6"
format_stamp: "Formatted at 2026-09-25 22:44:52 on dist-test-slave-5vdr"
server_key: "650c629a002fafdce07b55584ae7301c"
server_key_iv: "a7503e5c9704624c1a465fd5d79e4478"
server_key_version: "encryptionkey@0"
I20260925 22:44:52.311729 17343 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260925 22:44:52.314719 17358 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:44:52.316352 17343 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 22:44:52.316530 17343 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376259650642-11032-0/minicluster-data/ts-1/data,/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376259650642-11032-0/minicluster-data/ts-1/wal
uuid: "f5cc83f491e5409895f3eaf63bdc05c6"
format_stamp: "Formatted at 2026-09-25 22:44:52 on dist-test-slave-5vdr"
server_key: "650c629a002fafdce07b55584ae7301c"
server_key_iv: "a7503e5c9704624c1a465fd5d79e4478"
server_key_version: "encryptionkey@0"
I20260925 22:44:52.316679 17343 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376259650642-11032-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376259650642-11032-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376259650642-11032-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:44:52.343348 17343 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:44:52.346958 17343 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:44:52.347204 17343 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:44:52.347918 17343 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 22:44:52.348855 17343 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:44:52.348951 17343 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:44:52.349025 17343 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:44:52.349043 17343 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 22:44:52.359591 17343 rpc_server.cc:307] RPC server started. Bound to: 127.10.198.2:35073
I20260925 22:44:52.359622 17471 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.198.2:35073 every 8 connection(s)
I20260925 22:44:52.360713 17343 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376259650642-11032-0/minicluster-data/ts-1/data/info.pb
I20260925 22:44:52.369626 11032 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrWXm8X/build/debug/bin/kudu as pid 17343
I20260925 22:44:52.369777 11032 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376259650642-11032-0/minicluster-data/ts-1/wal/instance
I20260925 22:44:52.370175 11032 external_mini_cluster.cc:1428] Setting key 4f2648b02a0585f6ca517f7260cd1a36
Sep 25 22:44:52 dist-test-slave-5vdr krb5kdc[17102](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376292, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.198.2@KRBTEST.COM for oryx/127.10.198.62@KRBTEST.COM
I20260925 22:44:52.378372 17186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:44:52.362942 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.198.2:40421 (local address 127.10.198.62:41969)
0925 22:44:52.363117 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0925 22:44:52.363122 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:44:52.364256 (+ 1134us) server_negotiation.cc:407] Connection header received
0925 22:44:52.366031 (+ 1775us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:44:52.366035 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:44:52.366090 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:44:52.366164 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:44:52.367615 (+ 1451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:44:52.368339 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:44:52.369601 (+ 1262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:44:52.370062 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:44:52.370208 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:44:52.373528 (+ 3320us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:44:52.373535 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:44:52.373540 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:44:52.373590 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:44:52.375200 (+ 1610us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:44:52.376168 (+ 968us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:44:52.376176 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:44:52.376178 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:44:52.376310 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:44:52.376854 (+ 544us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:44:52.376856 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:44:52.376858 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:44:52.377089 (+ 231us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:44:52.377174 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0925 22:44:52.378067 (+ 893us) server_negotiation.cc:299] Negotiation successful
0925 22:44:52.378207 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260925 22:44:52.379913 17474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:44:52.363434 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.198.62:41969 (local address 127.10.198.2:40421)
0925 22:44:52.364061 (+ 627us) negotiation.cc:107] Waiting for socket to connect
0925 22:44:52.364119 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0925 22:44:52.365727 (+ 1608us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:44:52.366392 (+ 665us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:44:52.366400 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:44:52.366853 (+ 453us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:44:52.367399 (+ 546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:44:52.367415 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:44:52.368466 (+ 1051us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:44:52.368470 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:44:52.369441 (+ 971us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:44:52.369452 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:44:52.370326 (+ 874us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:44:52.370333 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:44:52.370526 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:44:52.371316 (+ 790us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:44:52.371339 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:44:52.373295 (+ 1956us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:44:52.375379 (+ 2084us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:44:52.375391 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:44:52.375418 (+ 27us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:44:52.376007 (+ 589us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:44:52.376440 (+ 433us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:44:52.376445 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:44:52.376450 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:44:52.376723 (+ 273us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:44:52.377224 (+ 501us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:44:52.377237 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:44:52.377703 (+ 466us) client_negotiation.cc:769] Sending connection context
0925 22:44:52.378024 (+ 321us) client_negotiation.cc:240] Negotiation successful
0925 22:44:52.378533 (+ 509us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":327,"thread_start_us":120,"threads_started":1}
I20260925 22:44:52.381469 17472 heartbeater.cc:347] Connected to a master server at 127.10.198.62:41969
I20260925 22:44:52.381870 17472 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for oryx/127.10.198.3@KRBTEST.COM; defaulting to no policy
I20260925 22:44:52.382537 17472 heartbeater.cc:511] Master 127.10.198.62:41969 requested a full tablet report, sending...
I20260925 22:44:52.383960 17147 ts_manager.cc:194] Registered new tserver with Master: f5cc83f491e5409895f3eaf63bdc05c6 (127.10.198.2:35073)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.10.198.3@KRBTEST.COM" created.
I20260925 22:44:52.384711 17147 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.10.198.2@KRBTEST.COM'} at 127.10.198.2:40421
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.10.198.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376259650642-11032-0/krb5kdc/oryx_127.10.198.3.keytab.
Entry for principal oryx/127.10.198.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376259650642-11032-0/krb5kdc/oryx_127.10.198.3.keytab.
I20260925 22:44:52.402705 11032 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrWXm8X/build/debug/bin/kudu
/tmp/dist-test-taskrWXm8X/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376259650642-11032-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376259650642-11032-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376259650642-11032-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376259650642-11032-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.198.3:0
--local_ip_for_outbound_sockets=127.10.198.3
--webserver_interface=127.10.198.3
--webserver_port=0
--tserver_master_addrs=127.10.198.62:41969
--keytab_file=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376259650642-11032-0/krb5kdc/oryx_127.10.198.3.keytab
--principal=oryx/127.10.198.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.198.20:44855
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376259650642-11032-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376259650642-11032-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376259650642-11032-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:44:52.580214 17477 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:44:52.580528 17477 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:44:52.580580 17477 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:44:52.585103 17477 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:44:52.585198 17477 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:44:52.585264 17477 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.198.3
I20260925 22:44:52.591477 17477 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.198.20:44855
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376259650642-11032-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376259650642-11032-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376259650642-11032-0/krb5kdc/oryx_127.10.198.3.keytab
--principal=oryx/127.10.198.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.198.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376259650642-11032-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.198.3
--webserver_port=0
--tserver_master_addrs=127.10.198.62:41969
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17477
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.198.3
--log_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376259650642-11032-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:09:16 UTC on ccycloud.quasar-afmhyw.root.comops.site
build id 4673379
I20260925 22:44:52.592778 17477 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:44:52.593905 17477 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:44:52.604746 17483 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:44:52.604746 17485 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:44:52.604763 17482 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:44:52.605511 17477 server_base.cc:1034] running on GCE node
I20260925 22:44:52.606290 17477 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:44:52.607347 17477 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:44:52.608582 17477 hybrid_clock.cc:648] HybridClock initialized: now 1790376292608546 us; error 64 us; skew 500 ppm
Sep 25 22:44:52 dist-test-slave-5vdr krb5kdc[17102](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376292, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.198.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:44:52.613480 17477 init.cc:377] Logged in from keytab as oryx/127.10.198.3@KRBTEST.COM (short username oryx)
I20260925 22:44:52.615180 17477 webserver.cc:460] Webserver started at http://127.10.198.3:41485/ using document root <none> and password file <none>
I20260925 22:44:52.616176 17477 fs_manager.cc:359] Metadata directory not provided
I20260925 22:44:52.616300 17477 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:44:52.616602 17477 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:44:52.620185 17477 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376259650642-11032-0/minicluster-data/ts-2/data/instance:
uuid: "1ffb9543e6c340f78a7f82ffff34faf9"
format_stamp: "Formatted at 2026-09-25 22:44:52 on dist-test-slave-5vdr"
server_key: "8afc29a4732d5cf1f2dbbf92e50d6c97"
server_key_iv: "50a1831e6afe996752ae3a138fe73551"
server_key_version: "encryptionkey@0"
I20260925 22:44:52.621097 17477 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376259650642-11032-0/minicluster-data/ts-2/wal/instance:
uuid: "1ffb9543e6c340f78a7f82ffff34faf9"
format_stamp: "Formatted at 2026-09-25 22:44:52 on dist-test-slave-5vdr"
server_key: "8afc29a4732d5cf1f2dbbf92e50d6c97"
server_key_iv: "50a1831e6afe996752ae3a138fe73551"
server_key_version: "encryptionkey@0"
I20260925 22:44:52.627683 17477 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260925 22:44:52.632099 17492 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:44:52.634116 17477 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260925 22:44:52.634330 17477 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376259650642-11032-0/minicluster-data/ts-2/data,/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376259650642-11032-0/minicluster-data/ts-2/wal
uuid: "1ffb9543e6c340f78a7f82ffff34faf9"
format_stamp: "Formatted at 2026-09-25 22:44:52 on dist-test-slave-5vdr"
server_key: "8afc29a4732d5cf1f2dbbf92e50d6c97"
server_key_iv: "50a1831e6afe996752ae3a138fe73551"
server_key_version: "encryptionkey@0"
I20260925 22:44:52.634519 17477 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376259650642-11032-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376259650642-11032-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376259650642-11032-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:44:52.669207 17477 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:44:52.672716 17477 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:44:52.672904 17477 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:44:52.673498 17477 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 22:44:52.674357 17477 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:44:52.674408 17477 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:44:52.674453 17477 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:44:52.674467 17477 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 22:44:52.684923 17477 rpc_server.cc:307] RPC server started. Bound to: 127.10.198.3:46715
I20260925 22:44:52.684949 17605 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.198.3:46715 every 8 connection(s)
I20260925 22:44:52.685954 17477 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376259650642-11032-0/minicluster-data/ts-2/data/info.pb
I20260925 22:44:52.692241 11032 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrWXm8X/build/debug/bin/kudu as pid 17477
I20260925 22:44:52.692400 11032 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376259650642-11032-0/minicluster-data/ts-2/wal/instance
I20260925 22:44:52.692744 11032 external_mini_cluster.cc:1428] Setting key a0d6038e590776dbd8f195b8cf2746bd
Sep 25 22:44:52 dist-test-slave-5vdr krb5kdc[17102](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376292, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.198.3@KRBTEST.COM for oryx/127.10.198.62@KRBTEST.COM
I20260925 22:44:52.703392 17186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:44:52.688313 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.198.3:35963 (local address 127.10.198.62:41969)
0925 22:44:52.688554 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0925 22:44:52.688558 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:44:52.689534 (+ 976us) server_negotiation.cc:407] Connection header received
0925 22:44:52.691209 (+ 1675us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:44:52.691212 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:44:52.691275 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:44:52.691374 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:44:52.692971 (+ 1597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:44:52.693796 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:44:52.695628 (+ 1832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:44:52.695969 (+ 341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:44:52.696070 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:44:52.699079 (+ 3009us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:44:52.699083 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:44:52.699085 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:44:52.699115 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:44:52.700910 (+ 1795us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:44:52.701669 (+ 759us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:44:52.701672 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:44:52.701673 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:44:52.701774 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:44:52.702224 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:44:52.702226 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:44:52.702228 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:44:52.702427 (+ 199us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:44:52.702513 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0925 22:44:52.703130 (+ 617us) server_negotiation.cc:299] Negotiation successful
0925 22:44:52.703261 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":99}
I20260925 22:44:52.704757 17608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:44:52.688722 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.198.62:41969 (local address 127.10.198.3:35963)
0925 22:44:52.689308 (+ 586us) negotiation.cc:107] Waiting for socket to connect
0925 22:44:52.689362 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0925 22:44:52.690881 (+ 1519us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:44:52.691578 (+ 697us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:44:52.691587 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:44:52.692090 (+ 503us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:44:52.692750 (+ 660us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:44:52.692769 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:44:52.693962 (+ 1193us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:44:52.693969 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:44:52.695479 (+ 1510us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:44:52.695492 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:44:52.696142 (+ 650us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:44:52.696145 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:44:52.696308 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:44:52.696983 (+ 675us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:44:52.697010 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:44:52.698811 (+ 1801us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:44:52.701150 (+ 2339us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:44:52.701157 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:44:52.701175 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:44:52.701526 (+ 351us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:44:52.701915 (+ 389us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:44:52.701918 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:44:52.701920 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:44:52.702121 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:44:52.702554 (+ 433us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:44:52.702561 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:44:52.702876 (+ 315us) client_negotiation.cc:769] Sending connection context
0925 22:44:52.703078 (+ 202us) client_negotiation.cc:240] Negotiation successful
0925 22:44:52.703436 (+ 358us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":368,"thread_start_us":150,"threads_started":1}
I20260925 22:44:52.706413 17606 heartbeater.cc:347] Connected to a master server at 127.10.198.62:41969
I20260925 22:44:52.706736 17606 heartbeater.cc:464] Registering TS with master...
I20260925 22:44:52.707406 17606 heartbeater.cc:511] Master 127.10.198.62:41969 requested a full tablet report, sending...
I20260925 22:44:52.708530 17147 ts_manager.cc:194] Registered new tserver with Master: 1ffb9543e6c340f78a7f82ffff34faf9 (127.10.198.3:46715)
I20260925 22:44:52.709168 17147 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.10.198.3@KRBTEST.COM'} at 127.10.198.3:35963
I20260925 22:44:52.719782 11032 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.10.198.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.10.198.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.10.198.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376259650642-11032-0/krb5kdc/oryx_127.10.198.61.keytab.
Entry for principal oryx/127.10.198.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376259650642-11032-0/krb5kdc/oryx_127.10.198.61.keytab.
I20260925 22:44:52.748728 11032 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrWXm8X/build/debug/bin/kudu
/tmp/dist-test-taskrWXm8X/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376259650642-11032-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376259650642-11032-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376259650642-11032-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376259650642-11032-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.198.61:39135
--webserver_interface=127.10.198.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376259650642-11032-0/krb5kdc/oryx_127.10.198.61.keytab
--principal=oryx/127.10.198.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.198.20:44855
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.198.62:41969,127.10.198.61:39135
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376259650642-11032-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376259650642-11032-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376259650642-11032-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:44:52.874343 17611 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:44:52.874670 17611 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:44:52.874712 17611 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:44:52.879390 17611 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 22:44:52.879480 17611 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:44:52.879501 17611 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:44:52.879519 17611 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 22:44:52.879535 17611 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 22:44:52.885716 17611 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.198.20:44855
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376259650642-11032-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376259650642-11032-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.10.198.62:41969,127.10.198.61:39135
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376259650642-11032-0/krb5kdc/oryx_127.10.198.61.keytab
--principal=oryx/127.10.198.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.198.61:39135
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376259650642-11032-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.10.198.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17611
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376259650642-11032-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:09:16 UTC on ccycloud.quasar-afmhyw.root.comops.site
build id 4673379
W20260925 22:44:52.885975 17611 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.10.198.62:41969,127.10.198.61:39135'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260925 22:44:52.886940 17611 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:44:52.888094 17611 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:44:52.897380 17619 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:44:52.897367 17617 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:44:52.897431 17611 server_base.cc:1034] running on GCE node
W20260925 22:44:52.897369 17616 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:44:52.898345 17611 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:44:52.899672 17611 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:44:52.900903 17611 hybrid_clock.cc:648] HybridClock initialized: now 1790376292900867 us; error 58 us; skew 500 ppm
Sep 25 22:44:52 dist-test-slave-5vdr krb5kdc[17102](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376292, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.198.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:44:52.904211 17611 init.cc:377] Logged in from keytab as oryx/127.10.198.61@KRBTEST.COM (short username oryx)
I20260925 22:44:52.905488 17611 webserver.cc:460] Webserver started at http://127.10.198.61:44807/ using document root <none> and password file <none>
I20260925 22:44:52.906284 17611 fs_manager.cc:359] Metadata directory not provided
I20260925 22:44:52.906395 17611 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:44:52.906636 17611 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:44:52.909125 17611 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376259650642-11032-0/minicluster-data/master-1/data/instance:
uuid: "f7d2a2767543443e8ba59b91bd97f4e1"
format_stamp: "Formatted at 2026-09-25 22:44:52 on dist-test-slave-5vdr"
server_key: "2768300dc9f8a3f83028d714d3778e90"
server_key_iv: "8695b7bbfd3640c7af056d5e01e738f8"
server_key_version: "encryptionkey@0"
I20260925 22:44:52.909806 17611 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376259650642-11032-0/minicluster-data/master-1/wal/instance:
uuid: "f7d2a2767543443e8ba59b91bd97f4e1"
format_stamp: "Formatted at 2026-09-25 22:44:52 on dist-test-slave-5vdr"
server_key: "2768300dc9f8a3f83028d714d3778e90"
server_key_iv: "8695b7bbfd3640c7af056d5e01e738f8"
server_key_version: "encryptionkey@0"
I20260925 22:44:52.914258 17611 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260925 22:44:52.917351 17626 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:44:52.918987 17611 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260925 22:44:52.919203 17611 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376259650642-11032-0/minicluster-data/master-1/data,/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376259650642-11032-0/minicluster-data/master-1/wal
uuid: "f7d2a2767543443e8ba59b91bd97f4e1"
format_stamp: "Formatted at 2026-09-25 22:44:52 on dist-test-slave-5vdr"
server_key: "2768300dc9f8a3f83028d714d3778e90"
server_key_iv: "8695b7bbfd3640c7af056d5e01e738f8"
server_key_version: "encryptionkey@0"
I20260925 22:44:52.919356 17611 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376259650642-11032-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376259650642-11032-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376259650642-11032-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:44:52.950954 17611 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:44:52.954216 17611 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:44:52.954469 17611 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:44:52.961958 17611 rpc_server.cc:307] RPC server started. Bound to: 127.10.198.61:39135
I20260925 22:44:52.962273 17678 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.198.61:39135 every 8 connection(s)
I20260925 22:44:52.963039 17611 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376259650642-11032-0/minicluster-data/master-1/data/info.pb
I20260925 22:44:52.966104 17679 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:44:52.966729 11032 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrWXm8X/build/debug/bin/kudu as pid 17611
I20260925 22:44:52.971338 17679 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.198.62" port: 41969 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 22:44:52 dist-test-slave-5vdr krb5kdc[17102](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376291, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.10.198.61@KRBTEST.COM
Sep 25 22:44:52 dist-test-slave-5vdr krb5kdc[17102](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376292, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.198.61@KRBTEST.COM for oryx/127.10.198.62@KRBTEST.COM
I20260925 22:44:52.984087 17682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:44:52.968495 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59206 (local address 127.10.198.61:39135)
0925 22:44:52.969036 (+ 541us) server_negotiation.cc:206] Beginning negotiation
0925 22:44:52.969053 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0925 22:44:52.969092 (+ 39us) server_negotiation.cc:407] Connection header received
0925 22:44:52.969871 (+ 779us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:44:52.969895 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:44:52.970277 (+ 382us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:44:52.970662 (+ 385us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:44:52.971541 (+ 879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:44:52.972720 (+ 1179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:44:52.973864 (+ 1144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:44:52.974535 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:44:52.974681 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:44:52.978391 (+ 3710us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:44:52.978397 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:44:52.978411 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:44:52.978444 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:44:52.980435 (+ 1991us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:44:52.980975 (+ 540us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:44:52.980982 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:44:52.980991 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:44:52.981116 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:44:52.981461 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:44:52.981466 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:44:52.981469 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:44:52.982009 (+ 540us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:44:52.982260 (+ 251us) server_negotiation.cc:1035] Waiting for connection context
0925 22:44:52.982590 (+ 330us) server_negotiation.cc:299] Negotiation successful
0925 22:44:52.983112 (+ 522us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":371,"thread_start_us":169,"threads_started":1}
I20260925 22:44:52.984932 17683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:44:52.973387 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.198.62:41969 (local address 127.0.0.1:42064)
0925 22:44:52.973919 (+ 532us) negotiation.cc:107] Waiting for socket to connect
0925 22:44:52.973970 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0925 22:44:52.974127 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:44:52.974436 (+ 309us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:44:52.974455 (+ 19us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:44:52.975010 (+ 555us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:44:52.975181 (+ 171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:44:52.975187 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:44:52.976125 (+ 938us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:44:52.976132 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:44:52.977015 (+ 883us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:44:52.977028 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:44:52.977892 (+ 864us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:44:52.977897 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:44:52.978022 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:44:52.978873 (+ 851us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:44:52.978888 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:44:52.980863 (+ 1975us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:44:52.983093 (+ 2230us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:44:52.983102 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:44:52.983109 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:44:52.983354 (+ 245us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:44:52.983678 (+ 324us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:44:52.983682 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:44:52.983684 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:44:52.983779 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:44:52.984385 (+ 606us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:44:52.984390 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:44:52.984498 (+ 108us) client_negotiation.cc:769] Sending connection context
0925 22:44:52.984623 (+ 125us) client_negotiation.cc:240] Negotiation successful
0925 22:44:52.984787 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":311,"thread_start_us":186,"threads_started":1}
I20260925 22:44:52.985023 17186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:44:52.973428 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42064 (local address 127.10.198.62:41969)
0925 22:44:52.973615 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0925 22:44:52.973621 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 22:44:52.974052 (+ 431us) server_negotiation.cc:407] Connection header received
0925 22:44:52.974230 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:44:52.974233 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:44:52.974289 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:44:52.974380 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:44:52.975365 (+ 985us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:44:52.975975 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:44:52.977266 (+ 1291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:44:52.977671 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:44:52.977811 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:44:52.981090 (+ 3279us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:44:52.981093 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:44:52.981095 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:44:52.981126 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:44:52.982953 (+ 1827us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:44:52.983469 (+ 516us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:44:52.983476 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:44:52.983480 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:44:52.983571 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:44:52.983906 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:44:52.983910 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:44:52.983913 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:44:52.984210 (+ 297us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:44:52.984331 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0925 22:44:52.984733 (+ 402us) server_negotiation.cc:299] Negotiation successful
0925 22:44:52.984869 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260925 22:44:52.986490 17679 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.198.61" port: 39135 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 22:44:52 dist-test-slave-5vdr krb5kdc[17102](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376292, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.198.61@KRBTEST.COM for oryx/127.10.198.61@KRBTEST.COM
I20260925 22:44:52.995597 17683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:44:52.987032 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.198.61:39135 (local address 127.0.0.1:59222)
0925 22:44:52.987168 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0925 22:44:52.987184 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 22:44:52.987257 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:44:52.987448 (+ 191us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:44:52.987450 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:44:52.987604 (+ 154us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:44:52.987704 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:44:52.987709 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:44:52.989443 (+ 1734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:44:52.989447 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:44:52.989977 (+ 530us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:44:52.989986 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:44:52.990453 (+ 467us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:44:52.990455 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:44:52.990529 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:44:52.991066 (+ 537us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:44:52.991078 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:44:52.992595 (+ 1517us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:44:52.994265 (+ 1670us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:44:52.994268 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:44:52.994270 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:44:52.994481 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:44:52.994735 (+ 254us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:44:52.994737 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:44:52.994739 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:44:52.994812 (+ 73us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:44:52.995166 (+ 354us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:44:52.995168 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:44:52.995249 (+ 81us) client_negotiation.cc:769] Sending connection context
0925 22:44:52.995332 (+ 83us) client_negotiation.cc:240] Negotiation successful
0925 22:44:52.995447 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":52}
I20260925 22:44:52.995625 17682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:44:52.987110 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59222 (local address 127.10.198.61:39135)
0925 22:44:52.987233 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0925 22:44:52.987237 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:44:52.987249 (+ 12us) server_negotiation.cc:407] Connection header received
0925 22:44:52.987317 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:44:52.987319 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:44:52.987353 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:44:52.987427 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:44:52.988052 (+ 625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:44:52.989265 (+ 1213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:44:52.990105 (+ 840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:44:52.990338 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:44:52.990428 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:44:52.992785 (+ 2357us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:44:52.992789 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:44:52.992791 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:44:52.992815 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:44:52.994145 (+ 1330us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:44:52.994578 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:44:52.994580 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:44:52.994581 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:44:52.994655 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:44:52.994889 (+ 234us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:44:52.994892 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:44:52.994893 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:44:52.995085 (+ 192us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:44:52.995165 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0925 22:44:52.995408 (+ 243us) server_negotiation.cc:299] Negotiation successful
0925 22:44:52.995506 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260925 22:44:52.999063 17679 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f7d2a2767543443e8ba59b91bd97f4e1: Bootstrap starting.
I20260925 22:44:53.001585 17679 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f7d2a2767543443e8ba59b91bd97f4e1: Neither blocks nor log segments found. Creating new log.
I20260925 22:44:53.002310 17679 log.cc:824] T 00000000000000000000000000000000 P f7d2a2767543443e8ba59b91bd97f4e1: Log is configured to *not* fsync() on all Append() calls
I20260925 22:44:53.004657 17679 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f7d2a2767543443e8ba59b91bd97f4e1: No bootstrap required, opened a new log
I20260925 22:44:53.008082 17679 raft_consensus.cc:348] T 00000000000000000000000000000000 P f7d2a2767543443e8ba59b91bd97f4e1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36158ad8c29246c2ab5424b4baf3038d" member_type: VOTER last_known_addr { host: "127.10.198.62" port: 41969 } } peers { permanent_uuid: "f7d2a2767543443e8ba59b91bd97f4e1" member_type: VOTER last_known_addr { host: "127.10.198.61" port: 39135 } }
I20260925 22:44:53.008324 17679 raft_consensus.cc:374] T 00000000000000000000000000000000 P f7d2a2767543443e8ba59b91bd97f4e1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:44:53.008358 17679 raft_consensus.cc:729] T 00000000000000000000000000000000 P f7d2a2767543443e8ba59b91bd97f4e1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f7d2a2767543443e8ba59b91bd97f4e1, State: Initialized, Role: FOLLOWER
I20260925 22:44:53.008781 17679 consensus_queue.cc:260] T 00000000000000000000000000000000 P f7d2a2767543443e8ba59b91bd97f4e1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36158ad8c29246c2ab5424b4baf3038d" member_type: VOTER last_known_addr { host: "127.10.198.62" port: 41969 } } peers { permanent_uuid: "f7d2a2767543443e8ba59b91bd97f4e1" member_type: VOTER last_known_addr { host: "127.10.198.61" port: 39135 } }
I20260925 22:44:53.009267 17685 sys_catalog.cc:455] T 00000000000000000000000000000000 P f7d2a2767543443e8ba59b91bd97f4e1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36158ad8c29246c2ab5424b4baf3038d" member_type: VOTER last_known_addr { host: "127.10.198.62" port: 41969 } } peers { permanent_uuid: "f7d2a2767543443e8ba59b91bd97f4e1" member_type: VOTER last_known_addr { host: "127.10.198.61" port: 39135 } } }
I20260925 22:44:53.009398 17685 sys_catalog.cc:458] T 00000000000000000000000000000000 P f7d2a2767543443e8ba59b91bd97f4e1 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 22:44:53.010070 17679 sys_catalog.cc:564] T 00000000000000000000000000000000 P f7d2a2767543443e8ba59b91bd97f4e1 [sys.catalog]: configured and running, proceeding with master startup.
W20260925 22:44:53.015424 17696 catalog_manager.cc:1558] T 00000000000000000000000000000000 P f7d2a2767543443e8ba59b91bd97f4e1: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 22:44:53.015563 17696 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 22:44:53.018188 17611 master_runner.cc:428] Detected that this master f7d2a2767543443e8ba59b91bd97f4e1 is joining an existing cluster
I20260925 22:44:53.018281 17611 master_runner.cc:432] Initiating AddMaster RPC to add 127.10.198.61:39135
I20260925 22:44:53.031945 17682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:44:53.021863 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59234 (local address 127.10.198.61:39135)
0925 22:44:53.021989 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0925 22:44:53.021994 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:44:53.022013 (+ 19us) server_negotiation.cc:407] Connection header received
0925 22:44:53.022061 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:44:53.022066 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:44:53.022115 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:44:53.022190 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:44:53.023157 (+ 967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:44:53.023995 (+ 838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:44:53.025181 (+ 1186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:44:53.025457 (+ 276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:44:53.025713 (+ 256us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:44:53.027563 (+ 1850us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:44:53.027565 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:44:53.027568 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:44:53.027591 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:44:53.029772 (+ 2181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:44:53.030294 (+ 522us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:44:53.030298 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:44:53.030300 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:44:53.030410 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:44:53.030758 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:44:53.030764 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:44:53.030768 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:44:53.031058 (+ 290us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:44:53.031191 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0925 22:44:53.031514 (+ 323us) server_negotiation.cc:299] Negotiation successful
0925 22:44:53.031731 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":70,"server-negotiator.queue_time_us":60}
I20260925 22:44:53.031975 17702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:44:53.021262 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.198.61:39135 (local address 127.0.0.1:59234)
0925 22:44:53.021629 (+ 367us) negotiation.cc:107] Waiting for socket to connect
0925 22:44:53.021656 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0925 22:44:53.021768 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:44:53.022280 (+ 512us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:44:53.022283 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:44:53.022631 (+ 348us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:44:53.022971 (+ 340us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:44:53.022982 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:44:53.024153 (+ 1171us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:44:53.024156 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:44:53.025079 (+ 923us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:44:53.025091 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:44:53.025724 (+ 633us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:44:53.025728 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:44:53.025854 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:44:53.026450 (+ 596us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:44:53.026464 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:44:53.026849 (+ 385us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:44:53.029914 (+ 3065us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:44:53.029917 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:44:53.029920 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:44:53.030171 (+ 251us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:44:53.030536 (+ 365us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:44:53.030540 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:44:53.030542 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:44:53.030644 (+ 102us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:44:53.031173 (+ 529us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:44:53.031176 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:44:53.031303 (+ 127us) client_negotiation.cc:769] Sending connection context
0925 22:44:53.031446 (+ 143us) client_negotiation.cc:240] Negotiation successful
0925 22:44:53.031650 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":263,"mutex_wait_us":67,"spinlock_wait_cycles":31616,"thread_start_us":143,"threads_started":1}
I20260925 22:44:53.033805 17186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:44:53.021295 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42066 (local address 127.10.198.62:41969)
0925 22:44:53.021435 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0925 22:44:53.021440 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:44:53.022118 (+ 678us) server_negotiation.cc:407] Connection header received
0925 22:44:53.022182 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:44:53.022186 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:44:53.022241 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:44:53.022331 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:44:53.024110 (+ 1779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:44:53.024991 (+ 881us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:44:53.027054 (+ 2063us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:44:53.027313 (+ 259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:44:53.027423 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:44:53.028980 (+ 1557us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:44:53.028985 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:44:53.028987 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:44:53.029020 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:44:53.030835 (+ 1815us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:44:53.031325 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:44:53.031330 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:44:53.031332 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:44:53.031444 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:44:53.031797 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:44:53.031802 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:44:53.031804 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:44:53.032352 (+ 548us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:44:53.032456 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0925 22:44:53.033497 (+ 1041us) server_negotiation.cc:299] Negotiation successful
0925 22:44:53.033643 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260925 22:44:53.033830 17703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:44:53.021282 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.198.62:41969 (local address 127.0.0.1:42066)
0925 22:44:53.021989 (+ 707us) negotiation.cc:107] Waiting for socket to connect
0925 22:44:53.022005 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 22:44:53.022108 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:44:53.022339 (+ 231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:44:53.022345 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:44:53.022631 (+ 286us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:44:53.023006 (+ 375us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:44:53.023014 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:44:53.025136 (+ 2122us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:44:53.025140 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:44:53.026884 (+ 1744us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:44:53.026897 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:44:53.027454 (+ 557us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:44:53.027458 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:44:53.027572 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:44:53.028179 (+ 607us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:44:53.028195 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:44:53.028740 (+ 545us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:44:53.030957 (+ 2217us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:44:53.030960 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:44:53.030962 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:44:53.031208 (+ 246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:44:53.031570 (+ 362us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:44:53.031573 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:44:53.031577 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:44:53.031669 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:44:53.032546 (+ 877us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:44:53.032551 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:44:53.032665 (+ 114us) client_negotiation.cc:769] Sending connection context
0925 22:44:53.033409 (+ 744us) client_negotiation.cc:240] Negotiation successful
0925 22:44:53.033608 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":653,"mutex_wait_us":110,"spinlock_wait_cycles":3456,"thread_start_us":127,"threads_started":1}
Sep 25 22:44:53 dist-test-slave-5vdr krb5kdc[17102](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376291, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.198.62@KRBTEST.COM for oryx/127.10.198.61@KRBTEST.COM
WARNING: no policy specified for oryx/127.10.198.60@KRBTEST.COM; defaulting to no policy
I20260925 22:44:53.051600 17682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:44:53.039629 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59244 (local address 127.10.198.61:39135)
0925 22:44:53.039768 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0925 22:44:53.039774 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 22:44:53.040880 (+ 1106us) server_negotiation.cc:407] Connection header received
0925 22:44:53.041137 (+ 257us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:44:53.041142 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:44:53.041208 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:44:53.041306 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:44:53.042166 (+ 860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:44:53.042966 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:44:53.044011 (+ 1045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:44:53.044318 (+ 307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:44:53.044425 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:44:53.047340 (+ 2915us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:44:53.047347 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:44:53.047351 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:44:53.047393 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:44:53.049444 (+ 2051us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:44:53.050135 (+ 691us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:44:53.050139 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:44:53.050141 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:44:53.050236 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:44:53.050572 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:44:53.050575 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:44:53.050576 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:44:53.050813 (+ 237us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:44:53.050912 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0925 22:44:53.051350 (+ 438us) server_negotiation.cc:299] Negotiation successful
0925 22:44:53.051469 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260925 22:44:53.051724 17704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:44:53.039614 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.198.61:39135 (local address 127.0.0.1:59244)
0925 22:44:53.040697 (+ 1083us) negotiation.cc:107] Waiting for socket to connect
0925 22:44:53.040768 (+ 71us) client_negotiation.cc:174] Beginning negotiation
0925 22:44:53.040960 (+ 192us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:44:53.041391 (+ 431us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:44:53.041407 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:44:53.041894 (+ 487us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:44:53.042029 (+ 135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:44:53.042036 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:44:53.043117 (+ 1081us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:44:53.043120 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:44:53.043802 (+ 682us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:44:53.043813 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:44:53.044459 (+ 646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:44:53.044462 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:44:53.044558 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:44:53.045218 (+ 660us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:44:53.045231 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:44:53.047116 (+ 1885us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:44:53.049621 (+ 2505us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:44:53.049635 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:44:53.049643 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:44:53.050000 (+ 357us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:44:53.050351 (+ 351us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:44:53.050354 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:44:53.050356 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:44:53.050458 (+ 102us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:44:53.050943 (+ 485us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:44:53.050952 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:44:53.051106 (+ 154us) client_negotiation.cc:769] Sending connection context
0925 22:44:53.051286 (+ 180us) client_negotiation.cc:240] Negotiation successful
0925 22:44:53.051527 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":870,"thread_start_us":93,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.10.198.60@KRBTEST.COM" created.
I20260925 22:44:53.053610 17147 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.10.198.61:39135: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "f7d2a2767543443e8ba59b91bd97f4e1"
member_type: NON_VOTER
last_known_addr {
host: "127.10.198.61"
port: 39135
}
attrs {
promote: true
}
}
dest_uuid: "36158ad8c29246c2ab5424b4baf3038d"
cas_config_opid_index: -1
I20260925 22:44:53.054445 17147 consensus_queue.cc:237] T 00000000000000000000000000000000 P 36158ad8c29246c2ab5424b4baf3038d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "36158ad8c29246c2ab5424b4baf3038d" member_type: VOTER last_known_addr { host: "127.10.198.62" port: 41969 } } peers { permanent_uuid: "f7d2a2767543443e8ba59b91bd97f4e1" member_type: NON_VOTER last_known_addr { host: "127.10.198.61" port: 39135 } attrs { promote: true } }
I20260925 22:44:53.056990 17709 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 36158ad8c29246c2ab5424b4baf3038d [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER f7d2a2767543443e8ba59b91bd97f4e1 (127.10.198.61) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "36158ad8c29246c2ab5424b4baf3038d" member_type: VOTER last_known_addr { host: "127.10.198.62" port: 41969 } } peers { permanent_uuid: "f7d2a2767543443e8ba59b91bd97f4e1" member_type: NON_VOTER last_known_addr { host: "127.10.198.61" port: 39135 } attrs { promote: true } } }
I20260925 22:44:53.057839 17653 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f7d2a2767543443e8ba59b91bd97f4e1 [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:44:53.059013 17709 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.10.198.61:39135
I20260925 22:44:53.059341 17709 sys_catalog.cc:455] T 00000000000000000000000000000000 P 36158ad8c29246c2ab5424b4baf3038d [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "36158ad8c29246c2ab5424b4baf3038d" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "36158ad8c29246c2ab5424b4baf3038d" member_type: VOTER last_known_addr { host: "127.10.198.62" port: 41969 } } peers { permanent_uuid: "f7d2a2767543443e8ba59b91bd97f4e1" member_type: NON_VOTER last_known_addr { host: "127.10.198.61" port: 39135 } attrs { promote: true } } }
I20260925 22:44:53.059561 17709 sys_catalog.cc:458] T 00000000000000000000000000000000 P 36158ad8c29246c2ab5424b4baf3038d [sys.catalog]: This master's current role is: LEADER
I20260925 22:44:53.059760 17653 raft_consensus.cc:1261] T 00000000000000000000000000000000 P f7d2a2767543443e8ba59b91bd97f4e1 [term 1 FOLLOWER]: Refusing update from remote peer 36158ad8c29246c2ab5424b4baf3038d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260925 22:44:53.060797 17611 master.cc:537] Master@127.10.198.61:39135 shutting down...
I20260925 22:44:53.060756 17709 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 36158ad8c29246c2ab5424b4baf3038d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f7d2a2767543443e8ba59b91bd97f4e1" member_type: NON_VOTER last_known_addr { host: "127.10.198.61" port: 39135 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
W20260925 22:44:53.063162 17131 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
W20260925 22:44:53.066385 17131 consensus_peers.cc:487] T 00000000000000000000000000000000 P 36158ad8c29246c2ab5424b4baf3038d -> Peer f7d2a2767543443e8ba59b91bd97f4e1 (127.10.198.61:39135): Couldn't send request to peer f7d2a2767543443e8ba59b91bd97f4e1. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260925 22:44:53.067073 17611 raft_consensus.cc:2229] T 00000000000000000000000000000000 P f7d2a2767543443e8ba59b91bd97f4e1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 22:44:53.067410 17611 raft_consensus.cc:2258] T 00000000000000000000000000000000 P f7d2a2767543443e8ba59b91bd97f4e1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 22:44:53.067530 17611 tablet_replica.cc:330] T 00000000000000000000000000000000 P f7d2a2767543443e8ba59b91bd97f4e1: stopping tablet replica
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.10.198.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376259650642-11032-0/krb5kdc/oryx_127.10.198.60.keytab.
Entry for principal oryx/127.10.198.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376259650642-11032-0/krb5kdc/oryx_127.10.198.60.keytab.
I20260925 22:44:53.072067 11032 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrWXm8X/build/debug/bin/kudu
/tmp/dist-test-taskrWXm8X/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376259650642-11032-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376259650642-11032-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376259650642-11032-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376259650642-11032-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.198.60:44389
--webserver_interface=127.10.198.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376259650642-11032-0/krb5kdc/oryx_127.10.198.60.keytab
--principal=oryx/127.10.198.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.198.20:44855
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.198.62:41969,127.10.198.61:39135,127.10.198.60:44389
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376259650642-11032-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376259650642-11032-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376259650642-11032-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 22:44:53.082671 17611 master.cc:552] Master@127.10.198.61:39135 shutdown complete.
I20260925 22:44:53.082760 17611 master_runner.cc:305] Clearing existing system tablet
I20260925 22:44:53.083026 17611 fs_manager.cc:359] Metadata directory not provided
I20260925 22:44:53.083104 17611 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:44:53.084367 17611 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 22:44:53.085675 17714 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:44:53.086009 17611 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376259650642-11032-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:44:53.086200 17611 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 22:44:53.086246 17611 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376259650642-11032-0/minicluster-data/master-1/data,/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376259650642-11032-0/minicluster-data/master-1/wal
uuid: "f7d2a2767543443e8ba59b91bd97f4e1"
format_stamp: "Formatted at 2026-09-25 22:44:52 on dist-test-slave-5vdr"
server_key: "2768300dc9f8a3f83028d714d3778e90"
server_key_iv: "8695b7bbfd3640c7af056d5e01e738f8"
server_key_version: "encryptionkey@0"
I20260925 22:44:53.088490 17611 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P f7d2a2767543443e8ba59b91bd97f4e1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 22:44:53.090873 17611 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P f7d2a2767543443e8ba59b91bd97f4e1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260925 22:44:53.091025 17611 log.cc:1196] T 00000000000000000000000000000000 P f7d2a2767543443e8ba59b91bd97f4e1: Deleting WAL directory at /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376259650642-11032-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260925 22:44:53.091341 17611 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P f7d2a2767543443e8ba59b91bd97f4e1: Deleting consensus metadata
I20260925 22:44:53.091576 17611 master_runner.cc:315] Copying system tablet from 127.10.198.62:41969
I20260925 22:44:53.093464 17611 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P f7d2a2767543443e8ba59b91bd97f4e1: tablet copy: Beginning tablet copy session from remote peer at address 127.10.198.62:41969
I20260925 22:44:53.102183 17186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:44:53.094434 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42082 (local address 127.10.198.62:41969)
0925 22:44:53.094583 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0925 22:44:53.094588 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:44:53.094817 (+ 229us) server_negotiation.cc:407] Connection header received
0925 22:44:53.094993 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:44:53.094996 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:44:53.095059 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:44:53.095153 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:44:53.095644 (+ 491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:44:53.096284 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:44:53.097248 (+ 964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:44:53.097477 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:44:53.097597 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:44:53.098855 (+ 1258us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:44:53.098858 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:44:53.098861 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:44:53.098897 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:44:53.100417 (+ 1520us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:44:53.100885 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:44:53.100889 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:44:53.100891 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:44:53.100980 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:44:53.101284 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:44:53.101289 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:44:53.101291 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:44:53.101493 (+ 202us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:44:53.101604 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0925 22:44:53.101867 (+ 263us) server_negotiation.cc:299] Negotiation successful
0925 22:44:53.101986 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260925 22:44:53.102193 17720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:44:53.094362 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.198.62:41969 (local address 127.0.0.1:42082)
0925 22:44:53.094714 (+ 352us) negotiation.cc:107] Waiting for socket to connect
0925 22:44:53.094727 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 22:44:53.094875 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:44:53.095172 (+ 297us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:44:53.095176 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:44:53.095395 (+ 219us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:44:53.095510 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:44:53.095516 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:44:53.096404 (+ 888us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:44:53.096410 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:44:53.097110 (+ 700us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:44:53.097120 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:44:53.097609 (+ 489us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:44:53.097612 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:44:53.097709 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:44:53.098306 (+ 597us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:44:53.098320 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:44:53.098685 (+ 365us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:44:53.100547 (+ 1862us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:44:53.100558 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:44:53.100561 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:44:53.100779 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:44:53.101094 (+ 315us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:44:53.101098 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:44:53.101100 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:44:53.101188 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:44:53.101596 (+ 408us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:44:53.101599 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:44:53.101704 (+ 105us) client_negotiation.cc:769] Sending connection context
0925 22:44:53.101816 (+ 112us) client_negotiation.cc:240] Negotiation successful
0925 22:44:53.101984 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":283,"thread_start_us":117,"threads_started":1}
I20260925 22:44:53.103179 17167 tablet_copy_service.cc:140] P 36158ad8c29246c2ab5424b4baf3038d: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer f7d2a2767543443e8ba59b91bd97f4e1 ({username='oryx', principal='oryx/127.10.198.61@KRBTEST.COM'} at 127.0.0.1:42082)
I20260925 22:44:53.103367 17167 tablet_copy_service.cc:161] P 36158ad8c29246c2ab5424b4baf3038d: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer f7d2a2767543443e8ba59b91bd97f4e1 at {username='oryx', principal='oryx/127.10.198.61@KRBTEST.COM'} at 127.0.0.1:42082: session id = f7d2a2767543443e8ba59b91bd97f4e1-00000000000000000000000000000000
I20260925 22:44:53.105265 17167 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 36158ad8c29246c2ab5424b4baf3038d: Tablet Copy: opened 0 blocks and 1 log segments
I20260925 22:44:53.107131 17611 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260925 22:44:53.109342 17611 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P f7d2a2767543443e8ba59b91bd97f4e1: tablet copy: Starting download of 0 data blocks...
I20260925 22:44:53.109589 17611 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P f7d2a2767543443e8ba59b91bd97f4e1: tablet copy: Starting download of 1 WAL segments...
I20260925 22:44:53.111794 17611 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P f7d2a2767543443e8ba59b91bd97f4e1: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260925 22:44:53.113873 17167 tablet_copy_service.cc:342] P 36158ad8c29246c2ab5424b4baf3038d: Request end of tablet copy session f7d2a2767543443e8ba59b91bd97f4e1-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.10.198.61@KRBTEST.COM'} at 127.0.0.1:42082
I20260925 22:44:53.114029 17167 tablet_copy_service.cc:433] P 36158ad8c29246c2ab5424b4baf3038d: ending tablet copy session f7d2a2767543443e8ba59b91bd97f4e1-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer f7d2a2767543443e8ba59b91bd97f4e1
I20260925 22:44:53.127306 17611 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:44:53.130043 17721 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:44:53.133410 17722 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:44:53.135062 17611 server_base.cc:1034] running on GCE node
W20260925 22:44:53.135201 17724 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:44:53.135437 17611 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:44:53.135836 17611 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:44:53.137090 17611 hybrid_clock.cc:648] HybridClock initialized: now 1790376293137077 us; error 41 us; skew 500 ppm
Sep 25 22:44:53 dist-test-slave-5vdr krb5kdc[17102](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376293, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.198.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:44:53.139336 17611 init.cc:377] Logged in from keytab as oryx/127.10.198.61@KRBTEST.COM (short username oryx)
I20260925 22:44:53.140095 17611 webserver.cc:460] Webserver started at http://127.10.198.61:39345/ using document root <none> and password file <none>
I20260925 22:44:53.140341 17611 fs_manager.cc:359] Metadata directory not provided
I20260925 22:44:53.140400 17611 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:44:53.141782 17611 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260925 22:44:53.142828 17731 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:44:53.143076 17611 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 22:44:53.143155 17611 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376259650642-11032-0/minicluster-data/master-1/data,/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376259650642-11032-0/minicluster-data/master-1/wal
uuid: "f7d2a2767543443e8ba59b91bd97f4e1"
format_stamp: "Formatted at 2026-09-25 22:44:52 on dist-test-slave-5vdr"
server_key: "2768300dc9f8a3f83028d714d3778e90"
server_key_iv: "8695b7bbfd3640c7af056d5e01e738f8"
server_key_version: "encryptionkey@0"
I20260925 22:44:53.143270 17611 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376259650642-11032-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376259650642-11032-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376259650642-11032-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:44:53.149295 17338 heartbeater.cc:503] Master 127.10.198.62:41969 was elected leader, sending a full tablet report...
I20260925 22:44:53.162936 17611 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:44:53.165405 17611 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:44:53.170646 17611 rpc_server.cc:307] RPC server started. Bound to: 127.10.198.61:39135
I20260925 22:44:53.170861 17783 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.198.61:39135 every 8 connection(s)
I20260925 22:44:53.171281 17611 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376259650642-11032-0/minicluster-data/master-1/data/info.pb
I20260925 22:44:53.172910 17784 sys_catalog.cc:263] Verifying existing consensus state
I20260925 22:44:53.173823 17784 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f7d2a2767543443e8ba59b91bd97f4e1: Bootstrap starting.
I20260925 22:44:53.183342 17784 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f7d2a2767543443e8ba59b91bd97f4e1: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 22:44:53.183753 17784 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f7d2a2767543443e8ba59b91bd97f4e1: Bootstrap complete.
I20260925 22:44:53.184361 17784 raft_consensus.cc:348] T 00000000000000000000000000000000 P f7d2a2767543443e8ba59b91bd97f4e1 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "36158ad8c29246c2ab5424b4baf3038d" member_type: VOTER last_known_addr { host: "127.10.198.62" port: 41969 } } peers { permanent_uuid: "f7d2a2767543443e8ba59b91bd97f4e1" member_type: NON_VOTER last_known_addr { host: "127.10.198.61" port: 39135 } attrs { promote: true } }
I20260925 22:44:53.184512 17784 raft_consensus.cc:729] T 00000000000000000000000000000000 P f7d2a2767543443e8ba59b91bd97f4e1 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: f7d2a2767543443e8ba59b91bd97f4e1, State: Initialized, Role: LEARNER
I20260925 22:44:53.184633 17784 consensus_queue.cc:260] T 00000000000000000000000000000000 P f7d2a2767543443e8ba59b91bd97f4e1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "36158ad8c29246c2ab5424b4baf3038d" member_type: VOTER last_known_addr { host: "127.10.198.62" port: 41969 } } peers { permanent_uuid: "f7d2a2767543443e8ba59b91bd97f4e1" member_type: NON_VOTER last_known_addr { host: "127.10.198.61" port: 39135 } attrs { promote: true } }
I20260925 22:44:53.185007 17787 sys_catalog.cc:455] T 00000000000000000000000000000000 P f7d2a2767543443e8ba59b91bd97f4e1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "36158ad8c29246c2ab5424b4baf3038d" member_type: VOTER last_known_addr { host: "127.10.198.62" port: 41969 } } peers { permanent_uuid: "f7d2a2767543443e8ba59b91bd97f4e1" member_type: NON_VOTER last_known_addr { host: "127.10.198.61" port: 39135 } attrs { promote: true } } }
I20260925 22:44:53.185179 17787 sys_catalog.cc:458] T 00000000000000000000000000000000 P f7d2a2767543443e8ba59b91bd97f4e1 [sys.catalog]: This master's current role is: LEARNER
I20260925 22:44:53.185206 17784 sys_catalog.cc:564] T 00000000000000000000000000000000 P f7d2a2767543443e8ba59b91bd97f4e1 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 22:44:53.189709 17798 catalog_manager.cc:1259] Loaded cluster ID: dec13579c22b435db994a21e4ce3e494
I20260925 22:44:53.189817 17798 catalog_manager.cc:1552] T 00000000000000000000000000000000 P f7d2a2767543443e8ba59b91bd97f4e1: loading cluster ID for follower catalog manager: success
I20260925 22:44:53.191519 17798 catalog_manager.cc:1574] T 00000000000000000000000000000000 P f7d2a2767543443e8ba59b91bd97f4e1: acquiring CA information for follower catalog manager: success
I20260925 22:44:53.192945 17798 catalog_manager.cc:1602] T 00000000000000000000000000000000 P f7d2a2767543443e8ba59b91bd97f4e1: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260925 22:44:53.248435 17712 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:44:53.248772 17712 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:44:53.248821 17712 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:44:53.253067 17712 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 22:44:53.253150 17712 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:44:53.253171 17712 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:44:53.253188 17712 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 22:44:53.253204 17712 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 22:44:53.258988 17712 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.198.20:44855
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376259650642-11032-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376259650642-11032-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.10.198.62:41969,127.10.198.61:39135,127.10.198.60:44389
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376259650642-11032-0/krb5kdc/oryx_127.10.198.60.keytab
--principal=oryx/127.10.198.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.198.60:44389
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376259650642-11032-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.10.198.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17712
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376259650642-11032-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:09:16 UTC on ccycloud.quasar-afmhyw.root.comops.site
build id 4673379
I20260925 22:44:53.260234 17712 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:44:53.261310 17712 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:44:53.271118 17803 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:44:53.271147 17806 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:44:53.271258 17712 server_base.cc:1034] running on GCE node
W20260925 22:44:53.271128 17804 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:44:53.272334 17712 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:44:53.273622 17712 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:44:53.274888 17712 hybrid_clock.cc:648] HybridClock initialized: now 1790376293274849 us; error 65 us; skew 500 ppm
Sep 25 22:44:53 dist-test-slave-5vdr krb5kdc[17102](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376293, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.198.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:44:53.278594 17712 init.cc:377] Logged in from keytab as oryx/127.10.198.60@KRBTEST.COM (short username oryx)
I20260925 22:44:53.279798 17712 webserver.cc:460] Webserver started at http://127.10.198.60:38459/ using document root <none> and password file <none>
I20260925 22:44:53.280611 17712 fs_manager.cc:359] Metadata directory not provided
I20260925 22:44:53.280699 17712 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:44:53.280951 17712 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:44:53.283337 17712 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376259650642-11032-0/minicluster-data/master-2/data/instance:
uuid: "ed073dd0b5e2464bbc701bc2862011ea"
format_stamp: "Formatted at 2026-09-25 22:44:53 on dist-test-slave-5vdr"
server_key: "ebbe0f68936d2d3a962b122958853d7c"
server_key_iv: "508c81ec409a1a6206635bddedf24e36"
server_key_version: "encryptionkey@0"
I20260925 22:44:53.284075 17712 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376259650642-11032-0/minicluster-data/master-2/wal/instance:
uuid: "ed073dd0b5e2464bbc701bc2862011ea"
format_stamp: "Formatted at 2026-09-25 22:44:53 on dist-test-slave-5vdr"
server_key: "ebbe0f68936d2d3a962b122958853d7c"
server_key_iv: "508c81ec409a1a6206635bddedf24e36"
server_key_version: "encryptionkey@0"
I20260925 22:44:53.289125 17712 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260925 22:44:53.292121 17813 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:44:53.293576 17712 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 22:44:53.293764 17712 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376259650642-11032-0/minicluster-data/master-2/data,/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376259650642-11032-0/minicluster-data/master-2/wal
uuid: "ed073dd0b5e2464bbc701bc2862011ea"
format_stamp: "Formatted at 2026-09-25 22:44:53 on dist-test-slave-5vdr"
server_key: "ebbe0f68936d2d3a962b122958853d7c"
server_key_iv: "508c81ec409a1a6206635bddedf24e36"
server_key_version: "encryptionkey@0"
I20260925 22:44:53.293911 17712 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376259650642-11032-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376259650642-11032-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376259650642-11032-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:44:53.309119 17712 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:44:53.312896 17712 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:44:53.313138 17712 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:44:53.320797 17712 rpc_server.cc:307] RPC server started. Bound to: 127.10.198.60:44389
I20260925 22:44:53.321138 17865 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.198.60:44389 every 8 connection(s)
I20260925 22:44:53.322058 17712 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376259650642-11032-0/minicluster-data/master-2/data/info.pb
I20260925 22:44:53.325381 17866 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:44:53.329831 17866 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.198.62" port: 41969 } has no permanent_uuid. Determining permanent_uuid...
I20260925 22:44:53.331750 11032 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrWXm8X/build/debug/bin/kudu as pid 17712
Sep 25 22:44:53 dist-test-slave-5vdr krb5kdc[17102](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376293, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.198.60@KRBTEST.COM for oryx/127.10.198.62@KRBTEST.COM
Sep 25 22:44:53 dist-test-slave-5vdr krb5kdc[17102](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376291, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.10.198.60@KRBTEST.COM
I20260925 22:44:53.346572 17186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:44:53.332329 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42096 (local address 127.10.198.62:41969)
0925 22:44:53.332512 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0925 22:44:53.332515 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 22:44:53.333539 (+ 1024us) server_negotiation.cc:407] Connection header received
0925 22:44:53.334554 (+ 1015us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:44:53.334558 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:44:53.334616 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:44:53.334720 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:44:53.336180 (+ 1460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:44:53.336723 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:44:53.337932 (+ 1209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:44:53.338169 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:44:53.338257 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:44:53.341706 (+ 3449us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:44:53.341712 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:44:53.341716 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:44:53.341754 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:44:53.343713 (+ 1959us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:44:53.344463 (+ 750us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:44:53.344469 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:44:53.344471 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:44:53.344617 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:44:53.345125 (+ 508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:44:53.345129 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:44:53.345131 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:44:53.345413 (+ 282us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:44:53.345495 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0925 22:44:53.346153 (+ 658us) server_negotiation.cc:299] Negotiation successful
0925 22:44:53.346319 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260925 22:44:53.347532 17868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:44:53.332779 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.198.62:41969 (local address 127.0.0.1:42096)
0925 22:44:53.333350 (+ 571us) negotiation.cc:107] Waiting for socket to connect
0925 22:44:53.333406 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0925 22:44:53.334290 (+ 884us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:44:53.335222 (+ 932us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:44:53.335226 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:44:53.335698 (+ 472us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:44:53.335974 (+ 276us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:44:53.335991 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:44:53.336897 (+ 906us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:44:53.336903 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:44:53.337795 (+ 892us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:44:53.337805 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:44:53.338347 (+ 542us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:44:53.338353 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:44:53.338478 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:44:53.339438 (+ 960us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:44:53.339458 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:44:53.341384 (+ 1926us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:44:53.343941 (+ 2557us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:44:53.343950 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:44:53.343965 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:44:53.344340 (+ 375us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:44:53.344735 (+ 395us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:44:53.344738 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:44:53.344739 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:44:53.344962 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:44:53.345516 (+ 554us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:44:53.345523 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:44:53.345846 (+ 323us) client_negotiation.cc:769] Sending connection context
0925 22:44:53.346027 (+ 181us) client_negotiation.cc:240] Negotiation successful
0925 22:44:53.346398 (+ 371us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":352,"mutex_wait_us":76,"spinlock_wait_cycles":25216,"thread_start_us":173,"threads_started":1}
I20260925 22:44:53.347532 17869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:44:53.332791 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45588 (local address 127.10.198.60:44389)
0925 22:44:53.333404 (+ 613us) server_negotiation.cc:206] Beginning negotiation
0925 22:44:53.333419 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0925 22:44:53.333457 (+ 38us) server_negotiation.cc:407] Connection header received
0925 22:44:53.334266 (+ 809us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:44:53.334290 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:44:53.334566 (+ 276us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:44:53.334767 (+ 201us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:44:53.335558 (+ 791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:44:53.336849 (+ 1291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:44:53.337756 (+ 907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:44:53.338255 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:44:53.338478 (+ 223us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:44:53.342063 (+ 3585us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:44:53.342072 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:44:53.342090 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:44:53.342120 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:44:53.344211 (+ 2091us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:44:53.344857 (+ 646us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:44:53.344864 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:44:53.344866 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:44:53.344956 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:44:53.345321 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:44:53.345324 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:44:53.345325 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:44:53.345845 (+ 520us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:44:53.346067 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0925 22:44:53.346374 (+ 307us) server_negotiation.cc:299] Negotiation successful
0925 22:44:53.346495 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":125,"server-negotiator.queue_time_us":507,"spinlock_wait_cycles":32512,"thread_start_us":194,"threads_started":1}
I20260925 22:44:53.349017 17866 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.198.61" port: 39135 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 22:44:53 dist-test-slave-5vdr krb5kdc[17102](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376293, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.198.60@KRBTEST.COM for oryx/127.10.198.61@KRBTEST.COM
I20260925 22:44:53.359221 17870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:44:53.349718 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59260 (local address 127.10.198.61:39135)
0925 22:44:53.350263 (+ 545us) server_negotiation.cc:206] Beginning negotiation
0925 22:44:53.350267 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:44:53.350281 (+ 14us) server_negotiation.cc:407] Connection header received
0925 22:44:53.350322 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:44:53.350324 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:44:53.350368 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:44:53.350478 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:44:53.350951 (+ 473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:44:53.351636 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:44:53.352583 (+ 947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:44:53.352822 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:44:53.352967 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:44:53.355412 (+ 2445us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:44:53.355416 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:44:53.355418 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:44:53.355448 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:44:53.357248 (+ 1800us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:44:53.357803 (+ 555us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:44:53.357808 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:44:53.357810 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:44:53.357918 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:44:53.358253 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:44:53.358256 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:44:53.358258 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:44:53.358476 (+ 218us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:44:53.358594 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0925 22:44:53.358909 (+ 315us) server_negotiation.cc:299] Negotiation successful
0925 22:44:53.359047 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":465,"thread_start_us":95,"threads_started":1}
I20260925 22:44:53.359220 17868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:44:53.349592 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.198.61:39135 (local address 127.0.0.1:59260)
0925 22:44:53.349726 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0925 22:44:53.349743 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 22:44:53.349826 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:44:53.350494 (+ 668us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:44:53.350498 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:44:53.350703 (+ 205us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:44:53.350815 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:44:53.350821 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:44:53.351770 (+ 949us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:44:53.351776 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:44:53.352416 (+ 640us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:44:53.352427 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:44:53.352973 (+ 546us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:44:53.352977 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:44:53.353090 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:44:53.353690 (+ 600us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:44:53.353706 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:44:53.355206 (+ 1500us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:44:53.357422 (+ 2216us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:44:53.357426 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:44:53.357428 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:44:53.357679 (+ 251us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:44:53.358035 (+ 356us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:44:53.358039 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:44:53.358041 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:44:53.358142 (+ 101us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:44:53.358593 (+ 451us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:44:53.358596 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:44:53.358710 (+ 114us) client_negotiation.cc:769] Sending connection context
0925 22:44:53.358834 (+ 124us) client_negotiation.cc:240] Negotiation successful
0925 22:44:53.359016 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":51}
I20260925 22:44:53.360960 17866 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.198.60" port: 44389 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 22:44:53 dist-test-slave-5vdr krb5kdc[17102](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376293, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.198.60@KRBTEST.COM for oryx/127.10.198.60@KRBTEST.COM
I20260925 22:44:53.370453 17869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:44:53.361626 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45604 (local address 127.10.198.60:44389)
0925 22:44:53.361747 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0925 22:44:53.361750 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 22:44:53.361833 (+ 83us) server_negotiation.cc:407] Connection header received
0925 22:44:53.361978 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:44:53.361981 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:44:53.362028 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:44:53.362097 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:44:53.362602 (+ 505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:44:53.363117 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:44:53.364262 (+ 1145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:44:53.364513 (+ 251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:44:53.364644 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:44:53.366865 (+ 2221us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:44:53.366883 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:44:53.366888 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:44:53.366942 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:44:53.368728 (+ 1786us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:44:53.369191 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:44:53.369198 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:44:53.369200 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:44:53.369279 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:44:53.369563 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:44:53.369568 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:44:53.369569 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:44:53.369761 (+ 192us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:44:53.369861 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0925 22:44:53.370100 (+ 239us) server_negotiation.cc:299] Negotiation successful
0925 22:44:53.370312 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":109,"server-negotiator.queue_time_us":42}
I20260925 22:44:53.370453 17868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:44:53.361588 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.198.60:44389 (local address 127.0.0.1:45604)
0925 22:44:53.361741 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0925 22:44:53.361760 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0925 22:44:53.361886 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:44:53.362141 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:44:53.362144 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:44:53.362357 (+ 213us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:44:53.362468 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:44:53.362478 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:44:53.363235 (+ 757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:44:53.363239 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:44:53.364130 (+ 891us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:44:53.364142 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:44:53.364644 (+ 502us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:44:53.364647 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:44:53.364713 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:44:53.365236 (+ 523us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:44:53.365248 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:44:53.366589 (+ 1341us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:44:53.368871 (+ 2282us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:44:53.368874 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:44:53.368876 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:44:53.369087 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:44:53.369368 (+ 281us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:44:53.369374 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:44:53.369376 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:44:53.369469 (+ 93us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:44:53.369858 (+ 389us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:44:53.369861 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:44:53.369948 (+ 87us) client_negotiation.cc:769] Sending connection context
0925 22:44:53.370058 (+ 110us) client_negotiation.cc:240] Negotiation successful
0925 22:44:53.370245 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":57,"mutex_wait_us":4}
I20260925 22:44:53.374940 17866 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ed073dd0b5e2464bbc701bc2862011ea: Bootstrap starting.
I20260925 22:44:53.378036 17866 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ed073dd0b5e2464bbc701bc2862011ea: Neither blocks nor log segments found. Creating new log.
I20260925 22:44:53.379300 17866 log.cc:824] T 00000000000000000000000000000000 P ed073dd0b5e2464bbc701bc2862011ea: Log is configured to *not* fsync() on all Append() calls
I20260925 22:44:53.382534 17866 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ed073dd0b5e2464bbc701bc2862011ea: No bootstrap required, opened a new log
I20260925 22:44:53.387130 17472 heartbeater.cc:503] Master 127.10.198.62:41969 was elected leader, sending a full tablet report...
I20260925 22:44:53.387264 17866 raft_consensus.cc:348] T 00000000000000000000000000000000 P ed073dd0b5e2464bbc701bc2862011ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36158ad8c29246c2ab5424b4baf3038d" member_type: VOTER last_known_addr { host: "127.10.198.62" port: 41969 } } peers { permanent_uuid: "f7d2a2767543443e8ba59b91bd97f4e1" member_type: VOTER last_known_addr { host: "127.10.198.61" port: 39135 } } peers { permanent_uuid: "ed073dd0b5e2464bbc701bc2862011ea" member_type: VOTER last_known_addr { host: "127.10.198.60" port: 44389 } }
I20260925 22:44:53.387638 17866 raft_consensus.cc:374] T 00000000000000000000000000000000 P ed073dd0b5e2464bbc701bc2862011ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:44:53.387715 17866 raft_consensus.cc:729] T 00000000000000000000000000000000 P ed073dd0b5e2464bbc701bc2862011ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ed073dd0b5e2464bbc701bc2862011ea, State: Initialized, Role: FOLLOWER
I20260925 22:44:53.388331 17866 consensus_queue.cc:260] T 00000000000000000000000000000000 P ed073dd0b5e2464bbc701bc2862011ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36158ad8c29246c2ab5424b4baf3038d" member_type: VOTER last_known_addr { host: "127.10.198.62" port: 41969 } } peers { permanent_uuid: "f7d2a2767543443e8ba59b91bd97f4e1" member_type: VOTER last_known_addr { host: "127.10.198.61" port: 39135 } } peers { permanent_uuid: "ed073dd0b5e2464bbc701bc2862011ea" member_type: VOTER last_known_addr { host: "127.10.198.60" port: 44389 } }
I20260925 22:44:53.388945 17872 sys_catalog.cc:455] T 00000000000000000000000000000000 P ed073dd0b5e2464bbc701bc2862011ea [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36158ad8c29246c2ab5424b4baf3038d" member_type: VOTER last_known_addr { host: "127.10.198.62" port: 41969 } } peers { permanent_uuid: "f7d2a2767543443e8ba59b91bd97f4e1" member_type: VOTER last_known_addr { host: "127.10.198.61" port: 39135 } } peers { permanent_uuid: "ed073dd0b5e2464bbc701bc2862011ea" member_type: VOTER last_known_addr { host: "127.10.198.60" port: 44389 } } }
I20260925 22:44:53.389101 17872 sys_catalog.cc:458] T 00000000000000000000000000000000 P ed073dd0b5e2464bbc701bc2862011ea [sys.catalog]: This master's current role is: FOLLOWER
I20260925 22:44:53.389925 17866 sys_catalog.cc:564] T 00000000000000000000000000000000 P ed073dd0b5e2464bbc701bc2862011ea [sys.catalog]: configured and running, proceeding with master startup.
W20260925 22:44:53.395169 17883 catalog_manager.cc:1558] T 00000000000000000000000000000000 P ed073dd0b5e2464bbc701bc2862011ea: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 22:44:53.395284 17883 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 22:44:53.400153 17712 master_runner.cc:428] Detected that this master ed073dd0b5e2464bbc701bc2862011ea is joining an existing cluster
I20260925 22:44:53.400252 17712 master_runner.cc:432] Initiating AddMaster RPC to add 127.10.198.60:44389
I20260925 22:44:53.401588 11032 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrWXm8X/build/debug/bin/kudu with pid 17209
I20260925 22:44:53.412086 11032 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrWXm8X/build/debug/bin/kudu
/tmp/dist-test-taskrWXm8X/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376259650642-11032-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376259650642-11032-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376259650642-11032-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376259650642-11032-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.198.1:41135
--local_ip_for_outbound_sockets=127.10.198.1
--tserver_master_addrs=127.10.198.62:41969
--webserver_port=35903
--webserver_interface=127.10.198.1
--keytab_file=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376259650642-11032-0/krb5kdc/oryx_127.10.198.1.keytab
--principal=oryx/127.10.198.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.198.20:44855
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.10.198.62:41969,127.10.198.61:39135
--tserver_master_addrs=127.10.198.62:41969,127.10.198.61:39135,127.10.198.60:44389 with env {KRB5CCNAME=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376259650642-11032-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376259650642-11032-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376259650642-11032-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 22:44:53.414528 17870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:44:53.403550 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59270 (local address 127.10.198.61:39135)
0925 22:44:53.403726 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0925 22:44:53.403732 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 22:44:53.404276 (+ 544us) server_negotiation.cc:407] Connection header received
0925 22:44:53.404442 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:44:53.404448 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:44:53.404515 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:44:53.404621 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:44:53.405330 (+ 709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:44:53.406308 (+ 978us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:44:53.407590 (+ 1282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:44:53.408058 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:44:53.408197 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:44:53.409966 (+ 1769us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:44:53.409974 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:44:53.409980 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:44:53.410018 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:44:53.412243 (+ 2225us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:44:53.412849 (+ 606us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:44:53.412856 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:44:53.412861 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:44:53.412963 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:44:53.413316 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:44:53.413322 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:44:53.413326 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:44:53.413596 (+ 270us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:44:53.413687 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0925 22:44:53.414168 (+ 481us) server_negotiation.cc:299] Negotiation successful
0925 22:44:53.414331 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260925 22:44:53.415179 17889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:44:53.403593 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.198.61:39135 (local address 127.0.0.1:59270)
0925 22:44:53.404160 (+ 567us) negotiation.cc:107] Waiting for socket to connect
0925 22:44:53.404178 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0925 22:44:53.404315 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:44:53.404733 (+ 418us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:44:53.404738 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:44:53.405017 (+ 279us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:44:53.405194 (+ 177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:44:53.405204 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:44:53.406471 (+ 1267us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:44:53.406475 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:44:53.407415 (+ 940us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:44:53.407428 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:44:53.408379 (+ 951us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:44:53.408383 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:44:53.408481 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:44:53.409158 (+ 677us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:44:53.409173 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:44:53.409786 (+ 613us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:44:53.412393 (+ 2607us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:44:53.412399 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:44:53.412404 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:44:53.412722 (+ 318us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:44:53.413089 (+ 367us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:44:53.413107 (+ 18us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:44:53.413112 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:44:53.413209 (+ 97us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:44:53.413764 (+ 555us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:44:53.413766 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:44:53.413888 (+ 122us) client_negotiation.cc:769] Sending connection context
0925 22:44:53.414780 (+ 892us) client_negotiation.cc:240] Negotiation successful
0925 22:44:53.414943 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":463,"mutex_wait_us":1,"thread_start_us":115,"threads_started":1}
I20260925 22:44:53.422314 17869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:44:53.410631 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45608 (local address 127.10.198.60:44389)
0925 22:44:53.410767 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0925 22:44:53.410771 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:44:53.413942 (+ 3171us) server_negotiation.cc:407] Connection header received
0925 22:44:53.414090 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:44:53.414093 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:44:53.414131 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:44:53.414177 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:44:53.414780 (+ 603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:44:53.415453 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:44:53.416685 (+ 1232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:44:53.417046 (+ 361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:44:53.417126 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:44:53.418635 (+ 1509us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:44:53.418639 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:44:53.418642 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:44:53.418670 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:44:53.420476 (+ 1806us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:44:53.420989 (+ 513us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:44:53.420993 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:44:53.420995 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:44:53.421077 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:44:53.421368 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:44:53.421371 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:44:53.421373 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:44:53.421588 (+ 215us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:44:53.421660 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0925 22:44:53.421984 (+ 324us) server_negotiation.cc:299] Negotiation successful
0925 22:44:53.422123 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260925 22:44:53.422376 17186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:44:53.408989 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42098 (local address 127.10.198.62:41969)
0925 22:44:53.409105 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0925 22:44:53.409110 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:44:53.409130 (+ 20us) server_negotiation.cc:407] Connection header received
0925 22:44:53.409181 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:44:53.409184 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:44:53.409234 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:44:53.409335 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:44:53.410418 (+ 1083us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:44:53.411272 (+ 854us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:44:53.413554 (+ 2282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:44:53.413948 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:44:53.414045 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:44:53.418106 (+ 4061us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:44:53.418111 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:44:53.418114 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:44:53.418150 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:44:53.420068 (+ 1918us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:44:53.420715 (+ 647us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:44:53.420722 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:44:53.420726 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:44:53.420826 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:44:53.421160 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:44:53.421166 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:44:53.421170 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:44:53.421437 (+ 267us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:44:53.421523 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0925 22:44:53.421970 (+ 447us) server_negotiation.cc:299] Negotiation successful
0925 22:44:53.422130 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260925 22:44:53.423168 17890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:44:53.408458 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.198.62:41969 (local address 127.0.0.1:42098)
0925 22:44:53.408955 (+ 497us) negotiation.cc:107] Waiting for socket to connect
0925 22:44:53.408975 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0925 22:44:53.409088 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:44:53.409386 (+ 298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:44:53.409389 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:44:53.409671 (+ 282us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:44:53.410208 (+ 537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:44:53.410220 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:44:53.411468 (+ 1248us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:44:53.411472 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:44:53.413363 (+ 1891us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:44:53.413377 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:44:53.414237 (+ 860us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:44:53.414242 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:44:53.414347 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:44:53.414983 (+ 636us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:44:53.415001 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:44:53.417892 (+ 2891us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:44:53.420216 (+ 2324us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:44:53.420223 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:44:53.420228 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:44:53.420576 (+ 348us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:44:53.420948 (+ 372us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:44:53.420954 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:44:53.420958 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:44:53.421056 (+ 98us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:44:53.421618 (+ 562us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:44:53.421624 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:44:53.421753 (+ 129us) client_negotiation.cc:769] Sending connection context
0925 22:44:53.422648 (+ 895us) client_negotiation.cc:240] Negotiation successful
0925 22:44:53.422847 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":387,"thread_start_us":183,"threads_started":1}
I20260925 22:44:53.423717 17891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:44:53.410375 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.198.60:44389 (local address 127.0.0.1:45608)
0925 22:44:53.413857 (+ 3482us) negotiation.cc:107] Waiting for socket to connect
0925 22:44:53.413876 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0925 22:44:53.413996 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:44:53.414297 (+ 301us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:44:53.414300 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:44:53.414500 (+ 200us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:44:53.414655 (+ 155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:44:53.414665 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:44:53.415558 (+ 893us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:44:53.415560 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:44:53.416519 (+ 959us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:44:53.416534 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:44:53.417324 (+ 790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:44:53.417327 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:44:53.417440 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:44:53.418027 (+ 587us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:44:53.418041 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:44:53.418473 (+ 432us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:44:53.420609 (+ 2136us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:44:53.420612 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:44:53.420614 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:44:53.420875 (+ 261us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:44:53.421182 (+ 307us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:44:53.421185 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:44:53.421187 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:44:53.421265 (+ 78us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:44:53.421723 (+ 458us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:44:53.421726 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:44:53.421803 (+ 77us) client_negotiation.cc:769] Sending connection context
0925 22:44:53.423000 (+ 1197us) client_negotiation.cc:240] Negotiation successful
0925 22:44:53.423159 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3392,"thread_start_us":343,"threads_started":1}
Sep 25 22:44:53 dist-test-slave-5vdr krb5kdc[17102](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376291, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.198.62@KRBTEST.COM for oryx/127.10.198.60@KRBTEST.COM
I20260925 22:44:53.460083 17704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:44:53.428903 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.198.60:44389 (local address 127.0.0.1:45618)
0925 22:44:53.429255 (+ 352us) negotiation.cc:107] Waiting for socket to connect
0925 22:44:53.429274 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0925 22:44:53.429386 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:44:53.429757 (+ 371us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:44:53.429763 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:44:53.430016 (+ 253us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:44:53.430172 (+ 156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:44:53.430184 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:44:53.431306 (+ 1122us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:44:53.431310 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:44:53.432407 (+ 1097us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:44:53.432421 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:44:53.433099 (+ 678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:44:53.433104 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:44:53.433211 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:44:53.433854 (+ 643us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:44:53.433870 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:44:53.435664 (+ 1794us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:44:53.458178 (+ 22514us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:44:53.458182 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:44:53.458184 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:44:53.458501 (+ 317us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:44:53.458873 (+ 372us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:44:53.458877 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:44:53.458879 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:44:53.458981 (+ 102us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:44:53.459475 (+ 494us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:44:53.459478 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:44:53.459579 (+ 101us) client_negotiation.cc:769] Sending connection context
0925 22:44:53.459665 (+ 86us) client_negotiation.cc:240] Negotiation successful
0925 22:44:53.459835 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":238}
I20260925 22:44:53.461184 17869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:44:53.429247 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45618 (local address 127.10.198.60:44389)
0925 22:44:53.429394 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0925 22:44:53.429399 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:44:53.429420 (+ 21us) server_negotiation.cc:407] Connection header received
0925 22:44:53.429522 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:44:53.429526 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:44:53.429595 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:44:53.429705 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:44:53.430341 (+ 636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:44:53.431143 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:44:53.432564 (+ 1421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:44:53.432938 (+ 374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:44:53.433076 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:44:53.435914 (+ 2838us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:44:53.435918 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:44:53.435922 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:44:53.435956 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:44:53.458003 (+ 22047us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:44:53.458659 (+ 656us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:44:53.458662 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:44:53.458664 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:44:53.458762 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:44:53.459102 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:44:53.459105 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:44:53.459107 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:44:53.459338 (+ 231us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:44:53.460777 (+ 1439us) server_negotiation.cc:1035] Waiting for connection context
0925 22:44:53.460878 (+ 101us) server_negotiation.cc:299] Negotiation successful
0925 22:44:53.461023 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260925 22:44:53.462214 17870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:44:53.455795 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59276 (local address 127.10.198.61:39135)
0925 22:44:53.455994 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0925 22:44:53.455999 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:44:53.456275 (+ 276us) server_negotiation.cc:407] Connection header received
0925 22:44:53.456944 (+ 669us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:44:53.456948 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:44:53.457292 (+ 344us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:44:53.457379 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:44:53.457917 (+ 538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:44:53.458711 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:44:53.460515 (+ 1804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:44:53.461226 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:44:53.461320 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:44:53.461491 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0925 22:44:53.461993 (+ 502us) server_negotiation.cc:299] Negotiation successful
0925 22:44:53.462084 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":108}
I20260925 22:44:53.462778 17893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:44:53.455694 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.198.61:39135 (local address 127.0.0.1:59276)
0925 22:44:53.456159 (+ 465us) negotiation.cc:107] Waiting for socket to connect
0925 22:44:53.456179 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0925 22:44:53.456747 (+ 568us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:44:53.457595 (+ 848us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:44:53.457598 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:44:53.457622 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:44:53.457750 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:44:53.457758 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:44:53.458909 (+ 1151us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:44:53.458912 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:44:53.460328 (+ 1416us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:44:53.460342 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:44:53.461618 (+ 1276us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:44:53.461622 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:44:53.461758 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:44:53.461830 (+ 72us) client_negotiation.cc:769] Sending connection context
0925 22:44:53.462386 (+ 556us) client_negotiation.cc:240] Negotiation successful
0925 22:44:53.462495 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":381,"thread_start_us":167,"threads_started":1}
I20260925 22:44:53.462740 17147 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.10.198.60:44389: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "ed073dd0b5e2464bbc701bc2862011ea"
member_type: NON_VOTER
last_known_addr {
host: "127.10.198.60"
port: 44389
}
attrs {
promote: true
}
}
dest_uuid: "36158ad8c29246c2ab5424b4baf3038d"
cas_config_opid_index: 5
I20260925 22:44:53.463150 17147 consensus_queue.cc:237] T 00000000000000000000000000000000 P 36158ad8c29246c2ab5424b4baf3038d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "36158ad8c29246c2ab5424b4baf3038d" member_type: VOTER last_known_addr { host: "127.10.198.62" port: 41969 } } peers { permanent_uuid: "f7d2a2767543443e8ba59b91bd97f4e1" member_type: NON_VOTER last_known_addr { host: "127.10.198.61" port: 39135 } attrs { promote: true } } peers { permanent_uuid: "ed073dd0b5e2464bbc701bc2862011ea" member_type: NON_VOTER last_known_addr { host: "127.10.198.60" port: 44389 } attrs { promote: true } }
I20260925 22:44:53.463620 17758 raft_consensus.cc:1203] T 00000000000000000000000000000000 P f7d2a2767543443e8ba59b91bd97f4e1 [term 1 LEARNER]: Deduplicated request from leader. Original: 0.0->[1.1-1.5] Dedup: 1.5->[]
I20260925 22:44:53.464253 17787 sys_catalog.cc:455] T 00000000000000000000000000000000 P f7d2a2767543443e8ba59b91bd97f4e1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 36158ad8c29246c2ab5424b4baf3038d. Latest consensus state: current_term: 1 leader_uuid: "36158ad8c29246c2ab5424b4baf3038d" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "36158ad8c29246c2ab5424b4baf3038d" member_type: VOTER last_known_addr { host: "127.10.198.62" port: 41969 } } peers { permanent_uuid: "f7d2a2767543443e8ba59b91bd97f4e1" member_type: NON_VOTER last_known_addr { host: "127.10.198.61" port: 39135 } attrs { promote: true } } }
I20260925 22:44:53.464388 17787 sys_catalog.cc:458] T 00000000000000000000000000000000 P f7d2a2767543443e8ba59b91bd97f4e1 [sys.catalog]: This master's current role is: LEARNER
I20260925 22:44:53.464429 17708 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 36158ad8c29246c2ab5424b4baf3038d [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER ed073dd0b5e2464bbc701bc2862011ea (127.10.198.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "36158ad8c29246c2ab5424b4baf3038d" member_type: VOTER last_known_addr { host: "127.10.198.62" port: 41969 } } peers { permanent_uuid: "f7d2a2767543443e8ba59b91bd97f4e1" member_type: NON_VOTER last_known_addr { host: "127.10.198.61" port: 39135 } attrs { promote: true } } peers { permanent_uuid: "ed073dd0b5e2464bbc701bc2862011ea" member_type: NON_VOTER last_known_addr { host: "127.10.198.60" port: 44389 } attrs { promote: true } } }
I20260925 22:44:53.465422 17758 raft_consensus.cc:1261] T 00000000000000000000000000000000 P f7d2a2767543443e8ba59b91bd97f4e1 [term 1 LEARNER]: Refusing update from remote peer 36158ad8c29246c2ab5424b4baf3038d: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260925 22:44:53.466250 17708 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.10.198.60:44389
I20260925 22:44:53.466480 17708 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 36158ad8c29246c2ab5424b4baf3038d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f7d2a2767543443e8ba59b91bd97f4e1" member_type: NON_VOTER last_known_addr { host: "127.10.198.61" port: 39135 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260925 22:44:53.466794 17708 sys_catalog.cc:455] T 00000000000000000000000000000000 P 36158ad8c29246c2ab5424b4baf3038d [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "36158ad8c29246c2ab5424b4baf3038d" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "36158ad8c29246c2ab5424b4baf3038d" member_type: VOTER last_known_addr { host: "127.10.198.62" port: 41969 } } peers { permanent_uuid: "f7d2a2767543443e8ba59b91bd97f4e1" member_type: NON_VOTER last_known_addr { host: "127.10.198.61" port: 39135 } attrs { promote: true } } peers { permanent_uuid: "ed073dd0b5e2464bbc701bc2862011ea" member_type: NON_VOTER last_known_addr { host: "127.10.198.60" port: 44389 } attrs { promote: true } } }
I20260925 22:44:53.466895 17708 sys_catalog.cc:458] T 00000000000000000000000000000000 P 36158ad8c29246c2ab5424b4baf3038d [sys.catalog]: This master's current role is: LEADER
I20260925 22:44:53.467645 17840 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ed073dd0b5e2464bbc701bc2862011ea [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:44:53.467824 17712 master.cc:537] Master@127.10.198.60:44389 shutting down...
I20260925 22:44:53.469517 17840 raft_consensus.cc:1261] T 00000000000000000000000000000000 P ed073dd0b5e2464bbc701bc2862011ea [term 1 FOLLOWER]: Refusing update from remote peer 36158ad8c29246c2ab5424b4baf3038d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260925 22:44:53.469209 17758 raft_consensus.cc:2932] T 00000000000000000000000000000000 P f7d2a2767543443e8ba59b91bd97f4e1 [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER ed073dd0b5e2464bbc701bc2862011ea (127.10.198.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "36158ad8c29246c2ab5424b4baf3038d" member_type: VOTER last_known_addr { host: "127.10.198.62" port: 41969 } } peers { permanent_uuid: "f7d2a2767543443e8ba59b91bd97f4e1" member_type: NON_VOTER last_known_addr { host: "127.10.198.61" port: 39135 } attrs { promote: true } } peers { permanent_uuid: "ed073dd0b5e2464bbc701bc2862011ea" member_type: NON_VOTER last_known_addr { host: "127.10.198.60" port: 44389 } attrs { promote: true } } }
I20260925 22:44:53.471557 17787 sys_catalog.cc:455] T 00000000000000000000000000000000 P f7d2a2767543443e8ba59b91bd97f4e1 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "36158ad8c29246c2ab5424b4baf3038d" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "36158ad8c29246c2ab5424b4baf3038d" member_type: VOTER last_known_addr { host: "127.10.198.62" port: 41969 } } peers { permanent_uuid: "f7d2a2767543443e8ba59b91bd97f4e1" member_type: NON_VOTER last_known_addr { host: "127.10.198.61" port: 39135 } attrs { promote: true } } peers { permanent_uuid: "ed073dd0b5e2464bbc701bc2862011ea" member_type: NON_VOTER last_known_addr { host: "127.10.198.60" port: 44389 } attrs { promote: true } } }
I20260925 22:44:53.471717 17787 sys_catalog.cc:458] T 00000000000000000000000000000000 P f7d2a2767543443e8ba59b91bd97f4e1 [sys.catalog]: This master's current role is: LEARNER
I20260925 22:44:53.473141 17712 raft_consensus.cc:2229] T 00000000000000000000000000000000 P ed073dd0b5e2464bbc701bc2862011ea [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 22:44:53.473274 17709 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 36158ad8c29246c2ab5424b4baf3038d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ed073dd0b5e2464bbc701bc2862011ea" member_type: NON_VOTER last_known_addr { host: "127.10.198.60" port: 44389 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 22:44:53.473470 17712 raft_consensus.cc:2258] T 00000000000000000000000000000000 P ed073dd0b5e2464bbc701bc2862011ea [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 22:44:53.473570 17712 tablet_replica.cc:330] T 00000000000000000000000000000000 P ed073dd0b5e2464bbc701bc2862011ea: stopping tablet replica
I20260925 22:44:53.474488 17709 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 36158ad8c29246c2ab5424b4baf3038d: attempting to promote NON_VOTER f7d2a2767543443e8ba59b91bd97f4e1 to VOTER
I20260925 22:44:53.474973 17709 consensus_queue.cc:237] T 00000000000000000000000000000000 P 36158ad8c29246c2ab5424b4baf3038d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "36158ad8c29246c2ab5424b4baf3038d" member_type: VOTER last_known_addr { host: "127.10.198.62" port: 41969 } } peers { permanent_uuid: "f7d2a2767543443e8ba59b91bd97f4e1" member_type: VOTER last_known_addr { host: "127.10.198.61" port: 39135 } attrs { promote: false } } peers { permanent_uuid: "ed073dd0b5e2464bbc701bc2862011ea" member_type: NON_VOTER last_known_addr { host: "127.10.198.60" port: 44389 } attrs { promote: true } }
I20260925 22:44:53.477002 17758 raft_consensus.cc:1261] T 00000000000000000000000000000000 P f7d2a2767543443e8ba59b91bd97f4e1 [term 1 LEARNER]: Refusing update from remote peer 36158ad8c29246c2ab5424b4baf3038d: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260925 22:44:53.477571 17708 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 36158ad8c29246c2ab5424b4baf3038d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f7d2a2767543443e8ba59b91bd97f4e1" member_type: VOTER last_known_addr { host: "127.10.198.61" port: 39135 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
W20260925 22:44:53.477897 17132 consensus_peers.cc:487] T 00000000000000000000000000000000 P 36158ad8c29246c2ab5424b4baf3038d -> Peer ed073dd0b5e2464bbc701bc2862011ea (127.10.198.60:44389): Couldn't send request to peer ed073dd0b5e2464bbc701bc2862011ea. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260925 22:44:53.479997 17708 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 36158ad8c29246c2ab5424b4baf3038d [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, f7d2a2767543443e8ba59b91bd97f4e1 (127.10.198.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "36158ad8c29246c2ab5424b4baf3038d" member_type: VOTER last_known_addr { host: "127.10.198.62" port: 41969 } } peers { permanent_uuid: "f7d2a2767543443e8ba59b91bd97f4e1" member_type: VOTER last_known_addr { host: "127.10.198.61" port: 39135 } attrs { promote: false } } peers { permanent_uuid: "ed073dd0b5e2464bbc701bc2862011ea" member_type: NON_VOTER last_known_addr { host: "127.10.198.60" port: 44389 } attrs { promote: true } } }
I20260925 22:44:53.481819 17708 sys_catalog.cc:455] T 00000000000000000000000000000000 P 36158ad8c29246c2ab5424b4baf3038d [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "36158ad8c29246c2ab5424b4baf3038d" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "36158ad8c29246c2ab5424b4baf3038d" member_type: VOTER last_known_addr { host: "127.10.198.62" port: 41969 } } peers { permanent_uuid: "f7d2a2767543443e8ba59b91bd97f4e1" member_type: VOTER last_known_addr { host: "127.10.198.61" port: 39135 } attrs { promote: false } } peers { permanent_uuid: "ed073dd0b5e2464bbc701bc2862011ea" member_type: NON_VOTER last_known_addr { host: "127.10.198.60" port: 44389 } attrs { promote: true } } }
I20260925 22:44:53.481976 17708 sys_catalog.cc:458] T 00000000000000000000000000000000 P 36158ad8c29246c2ab5424b4baf3038d [sys.catalog]: This master's current role is: LEADER
I20260925 22:44:53.482024 17712 master.cc:552] Master@127.10.198.60:44389 shutdown complete.
I20260925 22:44:53.482074 17712 master_runner.cc:305] Clearing existing system tablet
I20260925 22:44:53.482120 17708 sys_catalog.cc:455] T 00000000000000000000000000000000 P 36158ad8c29246c2ab5424b4baf3038d [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "36158ad8c29246c2ab5424b4baf3038d" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "36158ad8c29246c2ab5424b4baf3038d" member_type: VOTER last_known_addr { host: "127.10.198.62" port: 41969 } } peers { permanent_uuid: "f7d2a2767543443e8ba59b91bd97f4e1" member_type: VOTER last_known_addr { host: "127.10.198.61" port: 39135 } attrs { promote: false } } peers { permanent_uuid: "ed073dd0b5e2464bbc701bc2862011ea" member_type: NON_VOTER last_known_addr { host: "127.10.198.60" port: 44389 } attrs { promote: true } } }
I20260925 22:44:53.482195 17708 sys_catalog.cc:458] T 00000000000000000000000000000000 P 36158ad8c29246c2ab5424b4baf3038d [sys.catalog]: This master's current role is: LEADER
I20260925 22:44:53.482323 17712 fs_manager.cc:359] Metadata directory not provided
I20260925 22:44:53.482383 17712 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:44:53.483587 17758 raft_consensus.cc:2932] T 00000000000000000000000000000000 P f7d2a2767543443e8ba59b91bd97f4e1 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, f7d2a2767543443e8ba59b91bd97f4e1 (127.10.198.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "36158ad8c29246c2ab5424b4baf3038d" member_type: VOTER last_known_addr { host: "127.10.198.62" port: 41969 } } peers { permanent_uuid: "f7d2a2767543443e8ba59b91bd97f4e1" member_type: VOTER last_known_addr { host: "127.10.198.61" port: 39135 } attrs { promote: false } } peers { permanent_uuid: "ed073dd0b5e2464bbc701bc2862011ea" member_type: NON_VOTER last_known_addr { host: "127.10.198.60" port: 44389 } attrs { promote: true } } }
I20260925 22:44:53.485472 17787 sys_catalog.cc:455] T 00000000000000000000000000000000 P f7d2a2767543443e8ba59b91bd97f4e1 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "36158ad8c29246c2ab5424b4baf3038d" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "36158ad8c29246c2ab5424b4baf3038d" member_type: VOTER last_known_addr { host: "127.10.198.62" port: 41969 } } peers { permanent_uuid: "f7d2a2767543443e8ba59b91bd97f4e1" member_type: VOTER last_known_addr { host: "127.10.198.61" port: 39135 } attrs { promote: false } } peers { permanent_uuid: "ed073dd0b5e2464bbc701bc2862011ea" member_type: NON_VOTER last_known_addr { host: "127.10.198.60" port: 44389 } attrs { promote: true } } }
I20260925 22:44:53.485630 17787 sys_catalog.cc:458] T 00000000000000000000000000000000 P f7d2a2767543443e8ba59b91bd97f4e1 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 22:44:53.486557 17712 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.000s
I20260925 22:44:53.488054 17897 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:44:53.488406 17712 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376259650642-11032-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:44:53.488570 17712 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 22:44:53.488651 17712 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376259650642-11032-0/minicluster-data/master-2/data,/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376259650642-11032-0/minicluster-data/master-2/wal
uuid: "ed073dd0b5e2464bbc701bc2862011ea"
format_stamp: "Formatted at 2026-09-25 22:44:53 on dist-test-slave-5vdr"
server_key: "ebbe0f68936d2d3a962b122958853d7c"
server_key_iv: "508c81ec409a1a6206635bddedf24e36"
server_key_version: "encryptionkey@0"
I20260925 22:44:53.491268 17712 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P ed073dd0b5e2464bbc701bc2862011ea: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 22:44:53.494330 17712 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P ed073dd0b5e2464bbc701bc2862011ea: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260925 22:44:53.494467 17712 log.cc:1196] T 00000000000000000000000000000000 P ed073dd0b5e2464bbc701bc2862011ea: Deleting WAL directory at /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376259650642-11032-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260925 22:44:53.494742 17712 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P ed073dd0b5e2464bbc701bc2862011ea: Deleting consensus metadata
I20260925 22:44:53.494947 17712 master_runner.cc:315] Copying system tablet from 127.10.198.62:41969
I20260925 22:44:53.496369 17712 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P ed073dd0b5e2464bbc701bc2862011ea: tablet copy: Beginning tablet copy session from remote peer at address 127.10.198.62:41969
I20260925 22:44:53.510690 17186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:44:53.497194 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42106 (local address 127.10.198.62:41969)
0925 22:44:53.497349 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0925 22:44:53.497355 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 22:44:53.497671 (+ 316us) server_negotiation.cc:407] Connection header received
0925 22:44:53.497823 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:44:53.497826 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:44:53.497888 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:44:53.497987 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:44:53.498627 (+ 640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:44:53.499424 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:44:53.500803 (+ 1379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:44:53.501188 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:44:53.501307 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:44:53.506469 (+ 5162us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:44:53.506474 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:44:53.506478 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:44:53.506514 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:44:53.508492 (+ 1978us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:44:53.509178 (+ 686us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:44:53.509183 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:44:53.509184 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:44:53.509292 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:44:53.509648 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:44:53.509651 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:44:53.509653 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:44:53.509907 (+ 254us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:44:53.510020 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0925 22:44:53.510361 (+ 341us) server_negotiation.cc:299] Negotiation successful
0925 22:44:53.510527 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260925 22:44:53.510696 17903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:44:53.497063 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.198.62:41969 (local address 127.0.0.1:42106)
0925 22:44:53.497530 (+ 467us) negotiation.cc:107] Waiting for socket to connect
0925 22:44:53.497549 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0925 22:44:53.497703 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:44:53.498034 (+ 331us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:44:53.498037 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:44:53.498313 (+ 276us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:44:53.498469 (+ 156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:44:53.498478 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:44:53.499582 (+ 1104us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:44:53.499586 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:44:53.500637 (+ 1051us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:44:53.500653 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:44:53.501332 (+ 679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:44:53.501336 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:44:53.501449 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:44:53.505562 (+ 4113us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:44:53.505585 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:44:53.506185 (+ 600us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:44:53.508691 (+ 2506us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:44:53.508695 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:44:53.508698 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:44:53.509032 (+ 334us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:44:53.509415 (+ 383us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:44:53.509419 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:44:53.509421 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:44:53.509531 (+ 110us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:44:53.510029 (+ 498us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:44:53.510033 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:44:53.510156 (+ 123us) client_negotiation.cc:769] Sending connection context
0925 22:44:53.510289 (+ 133us) client_negotiation.cc:240] Negotiation successful
0925 22:44:53.510479 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":388,"thread_start_us":107,"threads_started":1}
I20260925 22:44:53.512393 17167 tablet_copy_service.cc:140] P 36158ad8c29246c2ab5424b4baf3038d: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer ed073dd0b5e2464bbc701bc2862011ea ({username='oryx', principal='oryx/127.10.198.60@KRBTEST.COM'} at 127.0.0.1:42106)
I20260925 22:44:53.512843 17167 tablet_copy_service.cc:161] P 36158ad8c29246c2ab5424b4baf3038d: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer ed073dd0b5e2464bbc701bc2862011ea at {username='oryx', principal='oryx/127.10.198.60@KRBTEST.COM'} at 127.0.0.1:42106: session id = ed073dd0b5e2464bbc701bc2862011ea-00000000000000000000000000000000
I20260925 22:44:53.514861 17167 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 36158ad8c29246c2ab5424b4baf3038d: Tablet Copy: opened 0 blocks and 1 log segments
I20260925 22:44:53.517115 17712 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260925 22:44:53.520365 17712 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P ed073dd0b5e2464bbc701bc2862011ea: tablet copy: Starting download of 0 data blocks...
I20260925 22:44:53.520821 17712 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P ed073dd0b5e2464bbc701bc2862011ea: tablet copy: Starting download of 1 WAL segments...
I20260925 22:44:53.522800 17712 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P ed073dd0b5e2464bbc701bc2862011ea: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260925 22:44:53.525273 17167 tablet_copy_service.cc:342] P 36158ad8c29246c2ab5424b4baf3038d: Request end of tablet copy session ed073dd0b5e2464bbc701bc2862011ea-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.10.198.60@KRBTEST.COM'} at 127.0.0.1:42106
I20260925 22:44:53.525401 17167 tablet_copy_service.cc:433] P 36158ad8c29246c2ab5424b4baf3038d: ending tablet copy session ed073dd0b5e2464bbc701bc2862011ea-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer ed073dd0b5e2464bbc701bc2862011ea
I20260925 22:44:53.534072 17712 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:44:53.536130 17904 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:44:53.537150 17905 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:44:53.537457 17907 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:44:53.541096 17712 server_base.cc:1034] running on GCE node
I20260925 22:44:53.541334 17712 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:44:53.541725 17712 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:44:53.542905 17712 hybrid_clock.cc:648] HybridClock initialized: now 1790376293542892 us; error 39 us; skew 500 ppm
Sep 25 22:44:53 dist-test-slave-5vdr krb5kdc[17102](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376293, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.198.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:44:53.545010 17712 init.cc:377] Logged in from keytab as oryx/127.10.198.60@KRBTEST.COM (short username oryx)
I20260925 22:44:53.545607 17712 webserver.cc:460] Webserver started at http://127.10.198.60:36669/ using document root <none> and password file <none>
I20260925 22:44:53.545836 17712 fs_manager.cc:359] Metadata directory not provided
I20260925 22:44:53.545897 17712 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:44:53.547569 17712 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 22:44:53.548916 17914 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:44:53.549353 17712 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260925 22:44:53.549445 17712 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376259650642-11032-0/minicluster-data/master-2/data,/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376259650642-11032-0/minicluster-data/master-2/wal
uuid: "ed073dd0b5e2464bbc701bc2862011ea"
format_stamp: "Formatted at 2026-09-25 22:44:53 on dist-test-slave-5vdr"
server_key: "ebbe0f68936d2d3a962b122958853d7c"
server_key_iv: "508c81ec409a1a6206635bddedf24e36"
server_key_version: "encryptionkey@0"
I20260925 22:44:53.549538 17712 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376259650642-11032-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376259650642-11032-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376259650642-11032-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:44:53.571143 17712 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:44:53.573804 17712 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:44:53.579053 17712 rpc_server.cc:307] RPC server started. Bound to: 127.10.198.60:44389
I20260925 22:44:53.579099 17969 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.198.60:44389 every 8 connection(s)
I20260925 22:44:53.579736 17712 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376259650642-11032-0/minicluster-data/master-2/data/info.pb
I20260925 22:44:53.581931 17970 sys_catalog.cc:263] Verifying existing consensus state
I20260925 22:44:53.583206 17970 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ed073dd0b5e2464bbc701bc2862011ea: Bootstrap starting.
W20260925 22:44:53.584110 17892 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:44:53.584457 17892 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:44:53.584538 17892 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:44:53.589939 17892 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:44:53.590052 17892 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:44:53.590166 17892 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.198.1
I20260925 22:44:53.596277 17892 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.198.20:44855
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376259650642-11032-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376259650642-11032-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376259650642-11032-0/krb5kdc/oryx_127.10.198.1.keytab
--principal=oryx/127.10.198.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.198.1:41135
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376259650642-11032-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.198.1
--webserver_port=35903
--tserver_master_addrs=127.10.198.62:41969,127.10.198.61:39135,127.10.198.60:44389
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17892
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.198.1
--log_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376259650642-11032-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:09:16 UTC on ccycloud.quasar-afmhyw.root.comops.site
build id 4673379
I20260925 22:44:53.597576 17892 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:44:53.597770 17970 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ed073dd0b5e2464bbc701bc2862011ea: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 22:44:53.598173 17970 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ed073dd0b5e2464bbc701bc2862011ea: Bootstrap complete.
I20260925 22:44:53.598826 17970 raft_consensus.cc:348] T 00000000000000000000000000000000 P ed073dd0b5e2464bbc701bc2862011ea [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "36158ad8c29246c2ab5424b4baf3038d" member_type: VOTER last_known_addr { host: "127.10.198.62" port: 41969 } } peers { permanent_uuid: "f7d2a2767543443e8ba59b91bd97f4e1" member_type: VOTER last_known_addr { host: "127.10.198.61" port: 39135 } attrs { promote: false } } peers { permanent_uuid: "ed073dd0b5e2464bbc701bc2862011ea" member_type: NON_VOTER last_known_addr { host: "127.10.198.60" port: 44389 } attrs { promote: true } }
I20260925 22:44:53.598955 17892 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 22:44:53.599033 17970 raft_consensus.cc:729] T 00000000000000000000000000000000 P ed073dd0b5e2464bbc701bc2862011ea [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: ed073dd0b5e2464bbc701bc2862011ea, State: Initialized, Role: LEARNER
I20260925 22:44:53.599169 17970 consensus_queue.cc:260] T 00000000000000000000000000000000 P ed073dd0b5e2464bbc701bc2862011ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "36158ad8c29246c2ab5424b4baf3038d" member_type: VOTER last_known_addr { host: "127.10.198.62" port: 41969 } } peers { permanent_uuid: "f7d2a2767543443e8ba59b91bd97f4e1" member_type: VOTER last_known_addr { host: "127.10.198.61" port: 39135 } attrs { promote: false } } peers { permanent_uuid: "ed073dd0b5e2464bbc701bc2862011ea" member_type: NON_VOTER last_known_addr { host: "127.10.198.60" port: 44389 } attrs { promote: true } }
I20260925 22:44:53.599578 17973 sys_catalog.cc:455] T 00000000000000000000000000000000 P ed073dd0b5e2464bbc701bc2862011ea [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "36158ad8c29246c2ab5424b4baf3038d" member_type: VOTER last_known_addr { host: "127.10.198.62" port: 41969 } } peers { permanent_uuid: "f7d2a2767543443e8ba59b91bd97f4e1" member_type: VOTER last_known_addr { host: "127.10.198.61" port: 39135 } attrs { promote: false } } peers { permanent_uuid: "ed073dd0b5e2464bbc701bc2862011ea" member_type: NON_VOTER last_known_addr { host: "127.10.198.60" port: 44389 } attrs { promote: true } } }
I20260925 22:44:53.599728 17970 sys_catalog.cc:564] T 00000000000000000000000000000000 P ed073dd0b5e2464bbc701bc2862011ea [sys.catalog]: configured and running, proceeding with master startup.
I20260925 22:44:53.599902 17973 sys_catalog.cc:458] T 00000000000000000000000000000000 P ed073dd0b5e2464bbc701bc2862011ea [sys.catalog]: This master's current role is: LEARNER
I20260925 22:44:53.604857 17984 catalog_manager.cc:1259] Loaded cluster ID: dec13579c22b435db994a21e4ce3e494
I20260925 22:44:53.605014 17984 catalog_manager.cc:1552] T 00000000000000000000000000000000 P ed073dd0b5e2464bbc701bc2862011ea: loading cluster ID for follower catalog manager: success
I20260925 22:44:53.607367 17984 catalog_manager.cc:1574] T 00000000000000000000000000000000 P ed073dd0b5e2464bbc701bc2862011ea: acquiring CA information for follower catalog manager: success
I20260925 22:44:53.609161 17984 catalog_manager.cc:1602] T 00000000000000000000000000000000 P ed073dd0b5e2464bbc701bc2862011ea: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260925 22:44:53.612430 17987 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:44:53.612432 17986 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:44:53.612447 17989 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:44:53.612635 17892 server_base.cc:1034] running on GCE node
I20260925 22:44:53.613286 17892 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:44:53.614111 17892 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:44:53.615337 17892 hybrid_clock.cc:648] HybridClock initialized: now 1790376293615294 us; error 63 us; skew 500 ppm
Sep 25 22:44:53 dist-test-slave-5vdr krb5kdc[17102](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376293, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.198.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:44:53.619202 17892 init.cc:377] Logged in from keytab as oryx/127.10.198.1@KRBTEST.COM (short username oryx)
I20260925 22:44:53.620603 17892 webserver.cc:460] Webserver started at http://127.10.198.1:35903/ using document root <none> and password file <none>
I20260925 22:44:53.621320 17892 fs_manager.cc:359] Metadata directory not provided
I20260925 22:44:53.621441 17892 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:44:53.625924 17892 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260925 22:44:53.628464 17996 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:44:53.629987 17892 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 22:44:53.630172 17892 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376259650642-11032-0/minicluster-data/ts-0/data,/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376259650642-11032-0/minicluster-data/ts-0/wal
uuid: "dcfd1608a17742d9a03fa549b12bb0c8"
format_stamp: "Formatted at 2026-09-25 22:44:52 on dist-test-slave-5vdr"
server_key: "945983d5c65a0dacba5b4f65b51478ca"
server_key_iv: "f34c5c64750b11fdda091a0914bba73f"
server_key_version: "encryptionkey@0"
I20260925 22:44:53.630811 17892 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376259650642-11032-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376259650642-11032-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376259650642-11032-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:44:53.652881 17892 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:44:53.656256 17892 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:44:53.656512 17892 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:44:53.657187 17892 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 22:44:53.658181 17892 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:44:53.658262 17892 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:44:53.658339 17892 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:44:53.658385 17892 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 22:44:53.672379 17892 rpc_server.cc:307] RPC server started. Bound to: 127.10.198.1:41135
I20260925 22:44:53.672398 18109 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.198.1:41135 every 8 connection(s)
I20260925 22:44:53.674021 17892 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376259650642-11032-0/minicluster-data/ts-0/data/info.pb
I20260925 22:44:53.680624 11032 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrWXm8X/build/debug/bin/kudu as pid 17892
I20260925 22:44:53.680843 11032 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrWXm8X/build/debug/bin/kudu with pid 17343
I20260925 22:44:53.689131 11032 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrWXm8X/build/debug/bin/kudu
/tmp/dist-test-taskrWXm8X/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376259650642-11032-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376259650642-11032-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376259650642-11032-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376259650642-11032-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.198.2:35073
--local_ip_for_outbound_sockets=127.10.198.2
--tserver_master_addrs=127.10.198.62:41969
--webserver_port=37641
--webserver_interface=127.10.198.2
--keytab_file=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376259650642-11032-0/krb5kdc/oryx_127.10.198.2.keytab
--principal=oryx/127.10.198.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.198.20:44855
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.10.198.62:41969,127.10.198.61:39135
--tserver_master_addrs=127.10.198.62:41969,127.10.198.61:39135,127.10.198.60:44389 with env {KRB5CCNAME=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376259650642-11032-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376259650642-11032-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376259650642-11032-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 25 22:44:53 dist-test-slave-5vdr krb5kdc[17102](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376293, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.198.1@KRBTEST.COM for oryx/127.10.198.62@KRBTEST.COM
Sep 25 22:44:53 dist-test-slave-5vdr krb5kdc[17102](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376293, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.198.1@KRBTEST.COM for oryx/127.10.198.61@KRBTEST.COM
Sep 25 22:44:53 dist-test-slave-5vdr krb5kdc[17102](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376293, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.198.1@KRBTEST.COM for oryx/127.10.198.60@KRBTEST.COM
I20260925 22:44:53.698973 17870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:44:53.678306 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.198.1:44623 (local address 127.10.198.61:39135)
0925 22:44:53.678463 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0925 22:44:53.678468 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:44:53.679151 (+ 683us) server_negotiation.cc:407] Connection header received
0925 22:44:53.679326 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:44:53.679329 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:44:53.679375 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:44:53.679451 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:44:53.681639 (+ 2188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:44:53.682501 (+ 862us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:44:53.684714 (+ 2213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:44:53.685021 (+ 307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:44:53.685112 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:44:53.693539 (+ 8427us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:44:53.693547 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:44:53.693551 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:44:53.693585 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:44:53.696687 (+ 3102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:44:53.697398 (+ 711us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:44:53.697402 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:44:53.697404 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:44:53.697502 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:44:53.697875 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:44:53.697879 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:44:53.697881 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:44:53.698137 (+ 256us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:44:53.698231 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0925 22:44:53.698606 (+ 375us) server_negotiation.cc:299] Negotiation successful
0925 22:44:53.698758 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260925 22:44:53.699028 17186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:44:53.676318 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.198.1:46649 (local address 127.10.198.62:41969)
0925 22:44:53.678363 (+ 2045us) server_negotiation.cc:206] Beginning negotiation
0925 22:44:53.678369 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 22:44:53.678388 (+ 19us) server_negotiation.cc:407] Connection header received
0925 22:44:53.679300 (+ 912us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:44:53.679304 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:44:53.679364 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:44:53.679451 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:44:53.681637 (+ 2186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:44:53.682455 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:44:53.684722 (+ 2267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:44:53.685176 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:44:53.685290 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:44:53.692273 (+ 6983us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:44:53.692280 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:44:53.692286 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:44:53.692325 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:44:53.695582 (+ 3257us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:44:53.696568 (+ 986us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:44:53.696574 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:44:53.696575 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:44:53.696692 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:44:53.697259 (+ 567us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:44:53.697266 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:44:53.697269 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:44:53.697573 (+ 304us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:44:53.697682 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0925 22:44:53.698618 (+ 936us) server_negotiation.cc:299] Negotiation successful
0925 22:44:53.698787 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1937}
I20260925 22:44:53.700703 18117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:44:53.677030 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.198.61:39135 (local address 127.10.198.1:44623)
0925 22:44:53.679045 (+ 2015us) negotiation.cc:107] Waiting for socket to connect
0925 22:44:53.679063 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0925 22:44:53.679230 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:44:53.679673 (+ 443us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:44:53.679685 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:44:53.680430 (+ 745us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:44:53.681419 (+ 989us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:44:53.681433 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:44:53.682705 (+ 1272us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:44:53.682708 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:44:53.684528 (+ 1820us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:44:53.684536 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:44:53.685345 (+ 809us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:44:53.685348 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:44:53.685587 (+ 239us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:44:53.689098 (+ 3511us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:44:53.689114 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:44:53.693269 (+ 4155us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:44:53.696917 (+ 3648us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:44:53.696922 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:44:53.696926 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:44:53.697250 (+ 324us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:44:53.697636 (+ 386us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:44:53.697643 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:44:53.697648 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:44:53.697743 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:44:53.698291 (+ 548us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:44:53.698294 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:44:53.698391 (+ 97us) client_negotiation.cc:769] Sending connection context
0925 22:44:53.699757 (+ 1366us) client_negotiation.cc:240] Negotiation successful
0925 22:44:53.699958 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1904,"mutex_wait_us":1534,"spinlock_wait_cycles":9984,"thread_start_us":382,"threads_started":1}
I20260925 22:44:53.700734 18114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:44:53.677027 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.198.62:41969 (local address 127.10.198.1:46649)
0925 22:44:53.677694 (+ 667us) negotiation.cc:107] Waiting for socket to connect
0925 22:44:53.677767 (+ 73us) client_negotiation.cc:174] Beginning negotiation
0925 22:44:53.679041 (+ 1274us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:44:53.679675 (+ 634us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:44:53.679686 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:44:53.680430 (+ 744us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:44:53.681401 (+ 971us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:44:53.681429 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:44:53.682700 (+ 1271us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:44:53.682704 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:44:53.684534 (+ 1830us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:44:53.684548 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:44:53.685492 (+ 944us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:44:53.685499 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:44:53.685702 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:44:53.689056 (+ 3354us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:44:53.689085 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:44:53.691998 (+ 2913us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:44:53.695782 (+ 3784us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:44:53.695793 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:44:53.695820 (+ 27us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:44:53.696397 (+ 577us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:44:53.696836 (+ 439us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:44:53.696840 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:44:53.696843 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:44:53.697111 (+ 268us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:44:53.697767 (+ 656us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:44:53.697777 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:44:53.698244 (+ 467us) client_negotiation.cc:769] Sending connection context
0925 22:44:53.699233 (+ 989us) client_negotiation.cc:240] Negotiation successful
0925 22:44:53.699638 (+ 405us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":385,"mutex_wait_us":307,"spinlock_wait_cycles":136448,"thread_start_us":173,"threads_started":1}
I20260925 22:44:53.701632 18116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:44:53.677504 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.198.1:53445 (local address 127.10.198.60:44389)
0925 22:44:53.677912 (+ 408us) server_negotiation.cc:206] Beginning negotiation
0925 22:44:53.677926 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0925 22:44:53.679009 (+ 1083us) server_negotiation.cc:407] Connection header received
0925 22:44:53.679624 (+ 615us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:44:53.679628 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:44:53.679681 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:44:53.679760 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:44:53.683126 (+ 3366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:44:53.684264 (+ 1138us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:44:53.689200 (+ 4936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:44:53.689603 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:44:53.689728 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:44:53.693044 (+ 3316us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:44:53.693049 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:44:53.693053 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:44:53.693087 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:44:53.695572 (+ 2485us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:44:53.697753 (+ 2181us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:44:53.697758 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:44:53.697761 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:44:53.697874 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:44:53.698208 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:44:53.698212 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:44:53.698213 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:44:53.698468 (+ 255us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:44:53.698570 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0925 22:44:53.701218 (+ 2648us) server_negotiation.cc:299] Negotiation successful
0925 22:44:53.701411 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":313,"thread_start_us":85,"threads_started":1}
I20260925 22:44:53.700996 18115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:44:53.677335 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.198.60:44389 (local address 127.10.198.1:53445)
0925 22:44:53.678859 (+ 1524us) negotiation.cc:107] Waiting for socket to connect
0925 22:44:53.678875 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 22:44:53.679052 (+ 177us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:44:53.679918 (+ 866us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:44:53.679921 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:44:53.682799 (+ 2878us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:44:53.682933 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:44:53.682942 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:44:53.684430 (+ 1488us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:44:53.684436 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:44:53.688860 (+ 4424us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:44:53.688880 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:44:53.689911 (+ 1031us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:44:53.689918 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:44:53.690035 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:44:53.690612 (+ 577us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:44:53.690634 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:44:53.692836 (+ 2202us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:44:53.697311 (+ 4475us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:44:53.697316 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:44:53.697318 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:44:53.697610 (+ 292us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:44:53.697996 (+ 386us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:44:53.698000 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:44:53.698002 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:44:53.698089 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:44:53.698617 (+ 528us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:44:53.698620 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:44:53.698708 (+ 88us) client_negotiation.cc:769] Sending connection context
0925 22:44:53.698833 (+ 125us) client_negotiation.cc:240] Negotiation successful
0925 22:44:53.699420 (+ 587us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1440,"mutex_wait_us":2,"thread_start_us":81,"threads_started":1}
I20260925 22:44:53.703451 18112 heartbeater.cc:347] Connected to a master server at 127.10.198.61:39135
I20260925 22:44:53.703896 18112 heartbeater.cc:464] Registering TS with master...
I20260925 22:44:53.704615 18112 heartbeater.cc:511] Master 127.10.198.61:39135 requested a full tablet report, sending...
I20260925 22:44:53.703451 18111 heartbeater.cc:347] Connected to a master server at 127.10.198.62:41969
I20260925 22:44:53.705513 18111 heartbeater.cc:464] Registering TS with master...
I20260925 22:44:53.705761 18111 heartbeater.cc:511] Master 127.10.198.62:41969 requested a full tablet report, sending...
I20260925 22:44:53.706488 18110 heartbeater.cc:347] Connected to a master server at 127.10.198.60:44389
I20260925 22:44:53.706602 18110 heartbeater.cc:464] Registering TS with master...
I20260925 22:44:53.706815 18110 heartbeater.cc:511] Master 127.10.198.60:44389 requested a full tablet report, sending...
I20260925 22:44:53.706826 17748 ts_manager.cc:194] Registered new tserver with Master: dcfd1608a17742d9a03fa549b12bb0c8 (127.10.198.1:41135)
I20260925 22:44:53.707010 17147 ts_manager.cc:194] Re-registered known tserver with Master: dcfd1608a17742d9a03fa549b12bb0c8 (127.10.198.1:41135)
I20260925 22:44:53.707775 17147 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.10.198.1@KRBTEST.COM'} at 127.10.198.1:46649
I20260925 22:44:53.709666 17931 ts_manager.cc:194] Registered new tserver with Master: dcfd1608a17742d9a03fa549b12bb0c8 (127.10.198.1:41135)
I20260925 22:44:53.711295 17606 heartbeater.cc:503] Master 127.10.198.62:41969 was elected leader, sending a full tablet report...
W20260925 22:44:53.838085 18118 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:44:53.838447 18118 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:44:53.838533 18118 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:44:53.843716 18118 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:44:53.843837 18118 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:44:53.843994 18118 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.198.2
I20260925 22:44:53.850764 18118 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.198.20:44855
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376259650642-11032-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376259650642-11032-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376259650642-11032-0/krb5kdc/oryx_127.10.198.2.keytab
--principal=oryx/127.10.198.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.198.2:35073
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376259650642-11032-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.198.2
--webserver_port=37641
--tserver_master_addrs=127.10.198.62:41969,127.10.198.61:39135,127.10.198.60:44389
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18118
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.198.2
--log_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376259650642-11032-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:09:16 UTC on ccycloud.quasar-afmhyw.root.comops.site
build id 4673379
I20260925 22:44:53.852125 18118 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:44:53.853262 18118 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:44:53.863793 18126 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:44:53.863798 18123 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:44:53.863793 18124 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:44:53.864389 18118 server_base.cc:1034] running on GCE node
I20260925 22:44:53.864890 18118 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:44:53.865643 18118 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:44:53.866832 18118 hybrid_clock.cc:648] HybridClock initialized: now 1790376293866818 us; error 44 us; skew 500 ppm
Sep 25 22:44:53 dist-test-slave-5vdr krb5kdc[17102](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376293, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.198.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:44:53.870247 18118 init.cc:377] Logged in from keytab as oryx/127.10.198.2@KRBTEST.COM (short username oryx)
I20260925 22:44:53.871413 18118 webserver.cc:460] Webserver started at http://127.10.198.2:37641/ using document root <none> and password file <none>
I20260925 22:44:53.872159 18118 fs_manager.cc:359] Metadata directory not provided
I20260925 22:44:53.872262 18118 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:44:53.876677 18118 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260925 22:44:53.879150 18133 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:44:53.880790 18118 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260925 22:44:53.880960 18118 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376259650642-11032-0/minicluster-data/ts-1/data,/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376259650642-11032-0/minicluster-data/ts-1/wal
uuid: "f5cc83f491e5409895f3eaf63bdc05c6"
format_stamp: "Formatted at 2026-09-25 22:44:52 on dist-test-slave-5vdr"
server_key: "650c629a002fafdce07b55584ae7301c"
server_key_iv: "a7503e5c9704624c1a465fd5d79e4478"
server_key_version: "encryptionkey@0"
I20260925 22:44:53.881585 18118 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376259650642-11032-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376259650642-11032-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376259650642-11032-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:44:53.906526 18118 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:44:53.910218 18118 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:44:53.910419 18118 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:44:53.911037 18118 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 22:44:53.912154 18118 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:44:53.912213 18118 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:44:53.912261 18118 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:44:53.912284 18118 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 22:44:53.922643 18118 rpc_server.cc:307] RPC server started. Bound to: 127.10.198.2:35073
I20260925 22:44:53.922658 18246 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.198.2:35073 every 8 connection(s)
I20260925 22:44:53.923893 18118 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376259650642-11032-0/minicluster-data/ts-1/data/info.pb
I20260925 22:44:53.928908 11032 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrWXm8X/build/debug/bin/kudu as pid 18118
I20260925 22:44:53.929103 11032 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrWXm8X/build/debug/bin/kudu with pid 17477
I20260925 22:44:53.936585 11032 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrWXm8X/build/debug/bin/kudu
/tmp/dist-test-taskrWXm8X/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376259650642-11032-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376259650642-11032-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376259650642-11032-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376259650642-11032-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.198.3:46715
--local_ip_for_outbound_sockets=127.10.198.3
--tserver_master_addrs=127.10.198.62:41969
--webserver_port=41485
--webserver_interface=127.10.198.3
--keytab_file=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376259650642-11032-0/krb5kdc/oryx_127.10.198.3.keytab
--principal=oryx/127.10.198.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.198.20:44855
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.10.198.62:41969,127.10.198.61:39135
--tserver_master_addrs=127.10.198.62:41969,127.10.198.61:39135,127.10.198.60:44389 with env {KRB5CCNAME=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376259650642-11032-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376259650642-11032-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376259650642-11032-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 25 22:44:53 dist-test-slave-5vdr krb5kdc[17102](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376293, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.198.2@KRBTEST.COM for oryx/127.10.198.60@KRBTEST.COM
Sep 25 22:44:53 dist-test-slave-5vdr krb5kdc[17102](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376293, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.198.2@KRBTEST.COM for oryx/127.10.198.62@KRBTEST.COM
Sep 25 22:44:53 dist-test-slave-5vdr krb5kdc[17102](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376293, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.198.2@KRBTEST.COM for oryx/127.10.198.61@KRBTEST.COM
I20260925 22:44:53.946892 17186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:44:53.925957 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.198.2:33805 (local address 127.10.198.62:41969)
0925 22:44:53.926113 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0925 22:44:53.926118 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:44:53.927630 (+ 1512us) server_negotiation.cc:407] Connection header received
0925 22:44:53.928945 (+ 1315us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:44:53.928951 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:44:53.929010 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:44:53.929084 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:44:53.931360 (+ 2276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:44:53.932288 (+ 928us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:44:53.934706 (+ 2418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:44:53.935074 (+ 368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:44:53.935170 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:44:53.941132 (+ 5962us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:44:53.941137 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:44:53.941140 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:44:53.941183 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:44:53.943745 (+ 2562us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:44:53.944592 (+ 847us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:44:53.944596 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:44:53.944598 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:44:53.944706 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:44:53.945255 (+ 549us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:44:53.945258 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:44:53.945260 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:44:53.945536 (+ 276us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:44:53.945682 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0925 22:44:53.946553 (+ 871us) server_negotiation.cc:299] Negotiation successful
0925 22:44:53.946711 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260925 22:44:53.946892 18116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:44:53.926131 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.198.2:35017 (local address 127.10.198.60:44389)
0925 22:44:53.926241 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0925 22:44:53.926244 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 22:44:53.927396 (+ 1152us) server_negotiation.cc:407] Connection header received
0925 22:44:53.928835 (+ 1439us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:44:53.928838 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:44:53.928904 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:44:53.928980 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:44:53.931944 (+ 2964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:44:53.932892 (+ 948us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:44:53.934697 (+ 1805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:44:53.935147 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:44:53.935255 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:44:53.941132 (+ 5877us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:44:53.941137 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:44:53.941140 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:44:53.941183 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:44:53.943424 (+ 2241us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:44:53.944596 (+ 1172us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:44:53.944600 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:44:53.944601 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:44:53.944706 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:44:53.945255 (+ 549us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:44:53.945258 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:44:53.945260 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:44:53.945536 (+ 276us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:44:53.945635 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0925 22:44:53.946553 (+ 918us) server_negotiation.cc:299] Negotiation successful
0925 22:44:53.946712 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260925 22:44:53.947571 17870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:44:53.928235 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.198.2:57051 (local address 127.10.198.61:39135)
0925 22:44:53.928367 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0925 22:44:53.928376 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 22:44:53.928483 (+ 107us) server_negotiation.cc:407] Connection header received
0925 22:44:53.928840 (+ 357us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:44:53.928844 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:44:53.928918 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:44:53.928981 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:44:53.931504 (+ 2523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:44:53.932296 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:44:53.934798 (+ 2502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:44:53.935157 (+ 359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:44:53.935227 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:44:53.941546 (+ 6319us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:44:53.941551 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:44:53.941555 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:44:53.941590 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:44:53.943550 (+ 1960us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:44:53.944606 (+ 1056us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:44:53.944613 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:44:53.944616 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:44:53.944728 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:44:53.945273 (+ 545us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:44:53.945279 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:44:53.945283 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:44:53.945577 (+ 294us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:44:53.945671 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0925 22:44:53.947215 (+ 1544us) server_negotiation.cc:299] Negotiation successful
0925 22:44:53.947373 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260925 22:44:53.948971 18253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:44:53.928059 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.198.61:39135 (local address 127.10.198.2:57051)
0925 22:44:53.928385 (+ 326us) negotiation.cc:107] Waiting for socket to connect
0925 22:44:53.928399 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 22:44:53.928620 (+ 221us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:44:53.929962 (+ 1342us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:44:53.929965 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:44:53.930216 (+ 251us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:44:53.931358 (+ 1142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:44:53.931366 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:44:53.932499 (+ 1133us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:44:53.932502 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:44:53.934513 (+ 2011us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:44:53.934529 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:44:53.935480 (+ 951us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:44:53.935484 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:44:53.936016 (+ 532us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:44:53.937136 (+ 1120us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:44:53.937148 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:44:53.941331 (+ 4183us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:44:53.943723 (+ 2392us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:44:53.943730 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:44:53.943733 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:44:53.944450 (+ 717us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:44:53.944868 (+ 418us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:44:53.944874 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:44:53.944879 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:44:53.945136 (+ 257us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:44:53.945762 (+ 626us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:44:53.945766 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:44:53.946195 (+ 429us) client_negotiation.cc:769] Sending connection context
0925 22:44:53.947799 (+ 1604us) client_negotiation.cc:240] Negotiation successful
0925 22:44:53.948034 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":262,"mutex_wait_us":998,"spinlock_wait_cycles":39936,"thread_start_us":145,"threads_started":1}
I20260925 22:44:53.948971 18251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:44:53.926538 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.198.60:44389 (local address 127.10.198.2:35017)
0925 22:44:53.927194 (+ 656us) negotiation.cc:107] Waiting for socket to connect
0925 22:44:53.927251 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0925 22:44:53.928570 (+ 1319us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:44:53.931306 (+ 2736us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:44:53.931313 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:44:53.931561 (+ 248us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:44:53.931735 (+ 174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:44:53.931745 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:44:53.933058 (+ 1313us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:44:53.933064 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:44:53.934521 (+ 1457us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:44:53.934537 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:44:53.935482 (+ 945us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:44:53.935488 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:44:53.936012 (+ 524us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:44:53.937144 (+ 1132us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:44:53.937166 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:44:53.940790 (+ 3624us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:44:53.943639 (+ 2849us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:44:53.943650 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:44:53.943678 (+ 28us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:44:53.944454 (+ 776us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:44:53.944843 (+ 389us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:44:53.944845 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:44:53.944847 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:44:53.945129 (+ 282us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:44:53.945708 (+ 579us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:44:53.945718 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:44:53.946192 (+ 474us) client_negotiation.cc:769] Sending connection context
0925 22:44:53.947183 (+ 991us) client_negotiation.cc:240] Negotiation successful
0925 22:44:53.947611 (+ 428us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":446,"mutex_wait_us":338,"spinlock_wait_cycles":22400,"thread_start_us":735,"threads_started":1}
I20260925 22:44:53.948971 18252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:44:53.926538 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.198.62:41969 (local address 127.10.198.2:33805)
0925 22:44:53.927535 (+ 997us) negotiation.cc:107] Waiting for socket to connect
0925 22:44:53.927548 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 22:44:53.928570 (+ 1022us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:44:53.929372 (+ 802us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:44:53.929385 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:44:53.930090 (+ 705us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:44:53.931124 (+ 1034us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:44:53.931152 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:44:53.932499 (+ 1347us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:44:53.932502 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:44:53.934525 (+ 2023us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:44:53.934530 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:44:53.935480 (+ 950us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:44:53.935484 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:44:53.936011 (+ 527us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:44:53.937056 (+ 1045us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:44:53.937132 (+ 76us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:44:53.940790 (+ 3658us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:44:53.943922 (+ 3132us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:44:53.943925 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:44:53.943928 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:44:53.944473 (+ 545us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:44:53.944840 (+ 367us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:44:53.944843 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:44:53.944845 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:44:53.945129 (+ 284us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:44:53.945732 (+ 603us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:44:53.945734 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:44:53.946192 (+ 458us) client_negotiation.cc:769] Sending connection context
0925 22:44:53.947536 (+ 1344us) client_negotiation.cc:240] Negotiation successful
0925 22:44:53.947905 (+ 369us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":907,"mutex_wait_us":602,"spinlock_wait_cycles":151296,"thread_start_us":211,"threads_started":1}
I20260925 22:44:53.950467 18247 heartbeater.cc:347] Connected to a master server at 127.10.198.60:44389
I20260925 22:44:53.950729 18247 heartbeater.cc:464] Registering TS with master...
I20260925 22:44:53.950871 18248 heartbeater.cc:347] Connected to a master server at 127.10.198.62:41969
I20260925 22:44:53.950946 18248 heartbeater.cc:464] Registering TS with master...
I20260925 22:44:53.951280 18249 heartbeater.cc:347] Connected to a master server at 127.10.198.61:39135
I20260925 22:44:53.951342 18249 heartbeater.cc:464] Registering TS with master...
I20260925 22:44:53.951469 18248 heartbeater.cc:511] Master 127.10.198.62:41969 requested a full tablet report, sending...
I20260925 22:44:53.951475 18247 heartbeater.cc:511] Master 127.10.198.60:44389 requested a full tablet report, sending...
I20260925 22:44:53.951479 18249 heartbeater.cc:511] Master 127.10.198.61:39135 requested a full tablet report, sending...
I20260925 22:44:53.952734 17147 ts_manager.cc:194] Re-registered known tserver with Master: f5cc83f491e5409895f3eaf63bdc05c6 (127.10.198.2:35073)
I20260925 22:44:53.952925 17931 ts_manager.cc:194] Registered new tserver with Master: f5cc83f491e5409895f3eaf63bdc05c6 (127.10.198.2:35073)
I20260925 22:44:53.953423 17748 ts_manager.cc:194] Registered new tserver with Master: f5cc83f491e5409895f3eaf63bdc05c6 (127.10.198.2:35073)
I20260925 22:44:53.953559 17147 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.10.198.2@KRBTEST.COM'} at 127.10.198.2:33805
I20260925 22:44:54.004483 18255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:44:54.000042 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.198.60:44389 (local address 127.0.0.1:45630)
0925 22:44:54.000298 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0925 22:44:54.000312 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 22:44:54.000427 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:44:54.000909 (+ 482us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:44:54.000911 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:44:54.000933 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:44:54.001033 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:44:54.001038 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:44:54.002094 (+ 1056us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:44:54.002097 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:44:54.002894 (+ 797us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:44:54.002904 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:44:54.003953 (+ 1049us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:44:54.003956 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:44:54.004065 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:44:54.004102 (+ 37us) client_negotiation.cc:769] Sending connection context
0925 22:44:54.004193 (+ 91us) client_negotiation.cc:240] Negotiation successful
0925 22:44:54.004294 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":89,"threads_started":1}
I20260925 22:44:54.004479 18116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:44:54.000072 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45630 (local address 127.10.198.60:44389)
0925 22:44:54.000264 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0925 22:44:54.000270 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 22:44:54.000403 (+ 133us) server_negotiation.cc:407] Connection header received
0925 22:44:54.000612 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:44:54.000618 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:44:54.000693 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:44:54.000848 (+ 155us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:44:54.001157 (+ 309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:44:54.001955 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:44:54.003049 (+ 1094us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:44:54.003738 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:44:54.003926 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:44:54.004120 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0925 22:44:54.004209 (+ 89us) server_negotiation.cc:299] Negotiation successful
0925 22:44:54.004324 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260925 22:44:54.005422 17941 raft_consensus.cc:1203] T 00000000000000000000000000000000 P ed073dd0b5e2464bbc701bc2862011ea [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20260925 22:44:54.006050 17973 sys_catalog.cc:455] T 00000000000000000000000000000000 P ed073dd0b5e2464bbc701bc2862011ea [sys.catalog]: SysCatalogTable state changed. Reason: New leader 36158ad8c29246c2ab5424b4baf3038d. Latest consensus state: current_term: 1 leader_uuid: "36158ad8c29246c2ab5424b4baf3038d" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "36158ad8c29246c2ab5424b4baf3038d" member_type: VOTER last_known_addr { host: "127.10.198.62" port: 41969 } } peers { permanent_uuid: "f7d2a2767543443e8ba59b91bd97f4e1" member_type: VOTER last_known_addr { host: "127.10.198.61" port: 39135 } attrs { promote: false } } peers { permanent_uuid: "ed073dd0b5e2464bbc701bc2862011ea" member_type: NON_VOTER last_known_addr { host: "127.10.198.60" port: 44389 } attrs { promote: true } } }
I20260925 22:44:54.006211 17973 sys_catalog.cc:458] T 00000000000000000000000000000000 P ed073dd0b5e2464bbc701bc2862011ea [sys.catalog]: This master's current role is: LEARNER
W20260925 22:44:54.067742 18254 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:44:54.068174 18254 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:44:54.068269 18254 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:44:54.073016 18254 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:44:54.073141 18254 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:44:54.073256 18254 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.198.3
I20260925 22:44:54.079216 18254 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.198.20:44855
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376259650642-11032-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376259650642-11032-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376259650642-11032-0/krb5kdc/oryx_127.10.198.3.keytab
--principal=oryx/127.10.198.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.198.3:46715
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376259650642-11032-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.198.3
--webserver_port=41485
--tserver_master_addrs=127.10.198.62:41969,127.10.198.61:39135,127.10.198.60:44389
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18254
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.198.3
--log_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376259650642-11032-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:09:16 UTC on ccycloud.quasar-afmhyw.root.comops.site
build id 4673379
I20260925 22:44:54.080560 18254 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:44:54.081704 18254 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:44:54.093003 18261 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:44:54.093005 18260 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:44:54.093178 18254 server_base.cc:1034] running on GCE node
W20260925 22:44:54.093005 18263 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:44:54.093837 18254 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:44:54.094672 18254 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:44:54.095961 18254 hybrid_clock.cc:648] HybridClock initialized: now 1790376294095903 us; error 68 us; skew 500 ppm
Sep 25 22:44:54 dist-test-slave-5vdr krb5kdc[17102](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376294, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.198.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:44:54.099197 18254 init.cc:377] Logged in from keytab as oryx/127.10.198.3@KRBTEST.COM (short username oryx)
I20260925 22:44:54.100452 18254 webserver.cc:460] Webserver started at http://127.10.198.3:41485/ using document root <none> and password file <none>
I20260925 22:44:54.101213 18254 fs_manager.cc:359] Metadata directory not provided
I20260925 22:44:54.101338 18254 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:44:54.106026 18254 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260925 22:44:54.108670 18270 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:44:54.110256 18254 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 22:44:54.110460 18254 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376259650642-11032-0/minicluster-data/ts-2/data,/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376259650642-11032-0/minicluster-data/ts-2/wal
uuid: "1ffb9543e6c340f78a7f82ffff34faf9"
format_stamp: "Formatted at 2026-09-25 22:44:52 on dist-test-slave-5vdr"
server_key: "8afc29a4732d5cf1f2dbbf92e50d6c97"
server_key_iv: "50a1831e6afe996752ae3a138fe73551"
server_key_version: "encryptionkey@0"
I20260925 22:44:54.111093 18254 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376259650642-11032-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376259650642-11032-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376259650642-11032-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:44:54.138592 18254 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:44:54.142577 18254 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:44:54.142825 18254 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:44:54.143556 18254 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 22:44:54.144770 18254 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:44:54.144851 18254 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:44:54.144927 18254 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:44:54.144990 18254 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 22:44:54.155378 18254 rpc_server.cc:307] RPC server started. Bound to: 127.10.198.3:46715
I20260925 22:44:54.155459 18383 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.198.3:46715 every 8 connection(s)
I20260925 22:44:54.156760 18254 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376259650642-11032-0/minicluster-data/ts-2/data/info.pb
I20260925 22:44:54.157644 11032 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrWXm8X/build/debug/bin/kudu as pid 18254
I20260925 22:44:54.173081 18394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:44:54.159818 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42110 (local address 127.10.198.62:41969)
0925 22:44:54.161184 (+ 1366us) server_negotiation.cc:206] Beginning negotiation
0925 22:44:54.161197 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0925 22:44:54.161575 (+ 378us) server_negotiation.cc:407] Connection header received
0925 22:44:54.161625 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:44:54.161628 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:44:54.161693 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:44:54.161786 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:44:54.162836 (+ 1050us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:44:54.163646 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:44:54.164942 (+ 1296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:44:54.165319 (+ 377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:44:54.165399 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:44:54.167833 (+ 2434us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:44:54.167862 (+ 29us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:44:54.167867 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:44:54.167902 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:44:54.170470 (+ 2568us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:44:54.171019 (+ 549us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:44:54.171040 (+ 21us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:44:54.171043 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:44:54.171146 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:44:54.171464 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:44:54.171469 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:44:54.171470 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:44:54.171718 (+ 248us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:44:54.171885 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0925 22:44:54.172692 (+ 807us) server_negotiation.cc:299] Negotiation successful
0925 22:44:54.172843 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1293,"thread_start_us":95,"threads_started":1}
I20260925 22:44:54.174970 18398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:44:54.160913 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45638 (local address 127.10.198.60:44389)
0925 22:44:54.162240 (+ 1327us) server_negotiation.cc:206] Beginning negotiation
0925 22:44:54.162245 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:44:54.162546 (+ 301us) server_negotiation.cc:407] Connection header received
0925 22:44:54.162594 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:44:54.162600 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:44:54.162659 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:44:54.162736 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:44:54.163791 (+ 1055us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:44:54.164585 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:44:54.166129 (+ 1544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:44:54.166487 (+ 358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:44:54.166558 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:44:54.170561 (+ 4003us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:44:54.170565 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:44:54.170568 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:44:54.170596 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:44:54.172588 (+ 1992us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:44:54.173106 (+ 518us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:44:54.173110 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:44:54.173112 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:44:54.173194 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:44:54.173718 (+ 524us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:44:54.173721 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:44:54.173723 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:44:54.173942 (+ 219us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:44:54.174050 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0925 22:44:54.174673 (+ 623us) server_negotiation.cc:299] Negotiation successful
0925 22:44:54.174802 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1242,"thread_start_us":99,"threads_started":1}
Sep 25 22:44:54 dist-test-slave-5vdr krb5kdc[17102](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376294, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.198.3@KRBTEST.COM for oryx/127.10.198.62@KRBTEST.COM
I20260925 22:44:54.177639 18399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:44:54.161638 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59292 (local address 127.10.198.61:39135)
0925 22:44:54.165802 (+ 4164us) server_negotiation.cc:206] Beginning negotiation
0925 22:44:54.165808 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 22:44:54.165823 (+ 15us) server_negotiation.cc:407] Connection header received
0925 22:44:54.165855 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:44:54.165858 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:44:54.165904 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:44:54.165997 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:44:54.167016 (+ 1019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:44:54.167772 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:44:54.169033 (+ 1261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:44:54.169451 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:44:54.169582 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:44:54.171249 (+ 1667us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:44:54.171252 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:44:54.171255 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:44:54.171283 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:44:54.173080 (+ 1797us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:44:54.173958 (+ 878us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:44:54.173962 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:44:54.173963 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:44:54.174060 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:44:54.174639 (+ 579us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:44:54.174642 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:44:54.174644 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:44:54.174871 (+ 227us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:44:54.175003 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0925 22:44:54.176816 (+ 1813us) server_negotiation.cc:299] Negotiation successful
0925 22:44:54.176985 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4077,"thread_start_us":92,"threads_started":1}
Sep 25 22:44:54 dist-test-slave-5vdr krb5kdc[17102](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376294, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.198.3@KRBTEST.COM for oryx/127.10.198.60@KRBTEST.COM
I20260925 22:44:54.178692 17147 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42110:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 22:44:54.182963 17147 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
Sep 25 22:44:54 dist-test-slave-5vdr krb5kdc[17102](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376294, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.198.3@KRBTEST.COM for oryx/127.10.198.61@KRBTEST.COM
I20260925 22:44:54.185141 18116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:44:54.159301 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.198.3:57989 (local address 127.10.198.60:44389)
0925 22:44:54.159539 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0925 22:44:54.159546 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 22:44:54.161305 (+ 1759us) server_negotiation.cc:407] Connection header received
0925 22:44:54.164552 (+ 3247us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:44:54.164555 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:44:54.164614 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:44:54.164668 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:44:54.165287 (+ 619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:44:54.166047 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:44:54.168807 (+ 2760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:44:54.169154 (+ 347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:44:54.169240 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:44:54.179534 (+ 10294us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:44:54.179551 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:44:54.179554 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:44:54.179588 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:44:54.181977 (+ 2389us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:44:54.182787 (+ 810us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:44:54.182790 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:44:54.182792 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:44:54.182897 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:44:54.183602 (+ 705us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:44:54.183606 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:44:54.183608 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:44:54.183876 (+ 268us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:44:54.183968 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0925 22:44:54.184833 (+ 865us) server_negotiation.cc:299] Negotiation successful
0925 22:44:54.184974 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":169}
I20260925 22:44:54.185140 17186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:44:54.159177 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.198.3:49227 (local address 127.10.198.62:41969)
0925 22:44:54.159443 (+ 266us) server_negotiation.cc:206] Beginning negotiation
0925 22:44:54.159449 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 22:44:54.161302 (+ 1853us) server_negotiation.cc:407] Connection header received
0925 22:44:54.162677 (+ 1375us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:44:54.162680 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:44:54.162728 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:44:54.162808 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:44:54.166638 (+ 3830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:44:54.167435 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:44:54.174421 (+ 6986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:44:54.174789 (+ 368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:44:54.174872 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:44:54.180004 (+ 5132us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:44:54.180009 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:44:54.180013 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:44:54.180057 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:44:54.181977 (+ 1920us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:44:54.182786 (+ 809us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:44:54.182790 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:44:54.182792 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:44:54.183024 (+ 232us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:44:54.183596 (+ 572us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:44:54.183600 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:44:54.183601 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:44:54.183876 (+ 275us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:44:54.183968 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0925 22:44:54.184833 (+ 865us) server_negotiation.cc:299] Negotiation successful
0925 22:44:54.184974 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":174}
I20260925 22:44:54.186683 17894 sys_catalog.cc:455] T 00000000000000000000000000000000 P 36158ad8c29246c2ab5424b4baf3038d [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "36158ad8c29246c2ab5424b4baf3038d" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "36158ad8c29246c2ab5424b4baf3038d" member_type: VOTER last_known_addr { host: "127.10.198.62" port: 41969 } } peers { permanent_uuid: "f7d2a2767543443e8ba59b91bd97f4e1" member_type: VOTER last_known_addr { host: "127.10.198.61" port: 39135 } attrs { promote: false } } peers { permanent_uuid: "ed073dd0b5e2464bbc701bc2862011ea" member_type: NON_VOTER last_known_addr { host: "127.10.198.60" port: 44389 } attrs { promote: true } } }
I20260925 22:44:54.186869 17894 sys_catalog.cc:458] T 00000000000000000000000000000000 P 36158ad8c29246c2ab5424b4baf3038d [sys.catalog]: This master's current role is: LEADER
I20260925 22:44:54.187320 18396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:44:54.159607 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.198.60:44389 (local address 127.10.198.3:57989)
0925 22:44:54.161132 (+ 1525us) negotiation.cc:107] Waiting for socket to connect
0925 22:44:54.161198 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0925 22:44:54.163298 (+ 2100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:44:54.164816 (+ 1518us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:44:54.164819 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:44:54.165027 (+ 208us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:44:54.165147 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:44:54.165154 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:44:54.166404 (+ 1250us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:44:54.166407 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:44:54.167975 (+ 1568us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:44:54.167988 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:44:54.173563 (+ 5575us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:44:54.173567 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:44:54.174019 (+ 452us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:44:54.175552 (+ 1533us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:44:54.175579 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:44:54.179289 (+ 3710us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:44:54.182376 (+ 3087us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:44:54.182379 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:44:54.182381 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:44:54.182666 (+ 285us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:44:54.183206 (+ 540us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:44:54.183209 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:44:54.183212 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:44:54.183467 (+ 255us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:44:54.184045 (+ 578us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:44:54.184054 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:44:54.184508 (+ 454us) client_negotiation.cc:769] Sending connection context
0925 22:44:54.185343 (+ 835us) client_negotiation.cc:240] Negotiation successful
0925 22:44:54.185836 (+ 493us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1339,"mutex_wait_us":349,"spinlock_wait_cycles":141952,"thread_start_us":164,"threads_started":1}
I20260925 22:44:54.189404 18393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:44:54.159471 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.198.62:41969 (local address 127.10.198.3:49227)
0925 22:44:54.161133 (+ 1662us) negotiation.cc:107] Waiting for socket to connect
0925 22:44:54.161198 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0925 22:44:54.162453 (+ 1255us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:44:54.163136 (+ 683us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:44:54.163146 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:44:54.163648 (+ 502us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:44:54.164542 (+ 894us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:44:54.164565 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:44:54.167586 (+ 3021us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:44:54.167589 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:44:54.173838 (+ 6249us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:44:54.173851 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:44:54.175052 (+ 1201us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:44:54.175059 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:44:54.175193 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:44:54.176227 (+ 1034us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:44:54.176242 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:44:54.179789 (+ 3547us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:44:54.182166 (+ 2377us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:44:54.182176 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:44:54.182204 (+ 28us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:44:54.182650 (+ 446us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:44:54.183206 (+ 556us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:44:54.183209 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:44:54.183212 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:44:54.183467 (+ 255us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:44:54.184045 (+ 578us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:44:54.184054 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:44:54.184508 (+ 454us) client_negotiation.cc:769] Sending connection context
0925 22:44:54.185827 (+ 1319us) client_negotiation.cc:240] Negotiation successful
0925 22:44:54.185959 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1411,"mutex_wait_us":75,"spinlock_wait_cycles":16128,"thread_start_us":164,"threads_started":1}
I20260925 22:44:54.189747 18384 heartbeater.cc:347] Connected to a master server at 127.10.198.60:44389
I20260925 22:44:54.190016 18384 heartbeater.cc:464] Registering TS with master...
I20260925 22:44:54.190681 18384 heartbeater.cc:511] Master 127.10.198.60:44389 requested a full tablet report, sending...
I20260925 22:44:54.191264 17894 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 36158ad8c29246c2ab5424b4baf3038d: attempting to promote NON_VOTER ed073dd0b5e2464bbc701bc2862011ea to VOTER
I20260925 22:44:54.191296 18385 heartbeater.cc:347] Connected to a master server at 127.10.198.62:41969
I20260925 22:44:54.191385 18385 heartbeater.cc:464] Registering TS with master...
I20260925 22:44:54.191592 18385 heartbeater.cc:511] Master 127.10.198.62:41969 requested a full tablet report, sending...
I20260925 22:44:54.191682 17894 consensus_queue.cc:237] T 00000000000000000000000000000000 P 36158ad8c29246c2ab5424b4baf3038d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "36158ad8c29246c2ab5424b4baf3038d" member_type: VOTER last_known_addr { host: "127.10.198.62" port: 41969 } } peers { permanent_uuid: "f7d2a2767543443e8ba59b91bd97f4e1" member_type: VOTER last_known_addr { host: "127.10.198.61" port: 39135 } attrs { promote: false } } peers { permanent_uuid: "ed073dd0b5e2464bbc701bc2862011ea" member_type: VOTER last_known_addr { host: "127.10.198.60" port: 44389 } attrs { promote: false } }
I20260925 22:44:54.192924 17931 ts_manager.cc:194] Registered new tserver with Master: 1ffb9543e6c340f78a7f82ffff34faf9 (127.10.198.3:46715)
I20260925 22:44:54.193150 17758 raft_consensus.cc:1261] T 00000000000000000000000000000000 P f7d2a2767543443e8ba59b91bd97f4e1 [term 1 FOLLOWER]: Refusing update from remote peer 36158ad8c29246c2ab5424b4baf3038d: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260925 22:44:54.193394 17146 ts_manager.cc:194] Re-registered known tserver with Master: 1ffb9543e6c340f78a7f82ffff34faf9 (127.10.198.3:46715)
I20260925 22:44:54.194150 17146 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.10.198.3@KRBTEST.COM'} at 127.10.198.3:49227
I20260925 22:44:54.195102 18392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:44:54.159472 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.198.61:39135 (local address 127.10.198.3:35131)
0925 22:44:54.162053 (+ 2581us) negotiation.cc:107] Waiting for socket to connect
0925 22:44:54.162069 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 22:44:54.162766 (+ 697us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:44:54.163202 (+ 436us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:44:54.163205 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:44:54.163961 (+ 756us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:44:54.164698 (+ 737us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:44:54.164706 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:44:54.165769 (+ 1063us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:44:54.165772 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:44:54.169139 (+ 3367us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:44:54.169150 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:44:54.169954 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:44:54.169958 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:44:54.170316 (+ 358us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:44:54.177126 (+ 6810us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:44:54.177141 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:44:54.185519 (+ 8378us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:44:54.188019 (+ 2500us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:44:54.188023 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:44:54.188026 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:44:54.188301 (+ 275us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:44:54.193780 (+ 5479us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:44:54.193783 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:44:54.193784 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:44:54.193912 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:44:54.194482 (+ 570us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:44:54.194485 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:44:54.194611 (+ 126us) client_negotiation.cc:769] Sending connection context
0925 22:44:54.194722 (+ 111us) client_negotiation.cc:240] Negotiation successful
0925 22:44:54.194912 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2481,"mutex_wait_us":2154,"thread_start_us":677,"threads_started":1}
I20260925 22:44:54.195596 17870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:44:54.159697 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.198.3:35131 (local address 127.10.198.61:39135)
0925 22:44:54.159829 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0925 22:44:54.159870 (+ 41us) server_negotiation.cc:399] Waiting for connection header
0925 22:44:54.162137 (+ 2267us) server_negotiation.cc:407] Connection header received
0925 22:44:54.162898 (+ 761us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:44:54.162902 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:44:54.162960 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:44:54.163021 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:44:54.164832 (+ 1811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:44:54.165583 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:44:54.169278 (+ 3695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:44:54.169607 (+ 329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:44:54.169700 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:44:54.185744 (+ 16044us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:44:54.185748 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:44:54.185751 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:44:54.185786 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:44:54.187836 (+ 2050us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:44:54.188450 (+ 614us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:44:54.188453 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:44:54.188455 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:44:54.188580 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:44:54.194058 (+ 5478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:44:54.194062 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:44:54.194064 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:44:54.194317 (+ 253us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:44:54.194429 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0925 22:44:54.195341 (+ 912us) server_negotiation.cc:299] Negotiation successful
0925 22:44:54.195480 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260925 22:44:54.196053 18402 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 36158ad8c29246c2ab5424b4baf3038d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f7d2a2767543443e8ba59b91bd97f4e1" member_type: VOTER last_known_addr { host: "127.10.198.61" port: 39135 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260925 22:44:54.196106 17941 raft_consensus.cc:1261] T 00000000000000000000000000000000 P ed073dd0b5e2464bbc701bc2862011ea [term 1 LEARNER]: Refusing update from remote peer 36158ad8c29246c2ab5424b4baf3038d: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260925 22:44:54.196563 18402 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 36158ad8c29246c2ab5424b4baf3038d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ed073dd0b5e2464bbc701bc2862011ea" member_type: VOTER last_known_addr { host: "127.10.198.60" port: 44389 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260925 22:44:54.198074 18386 heartbeater.cc:347] Connected to a master server at 127.10.198.61:39135
I20260925 22:44:54.198388 18386 heartbeater.cc:464] Registering TS with master...
I20260925 22:44:54.198659 18386 heartbeater.cc:511] Master 127.10.198.61:39135 requested a full tablet report, sending...
I20260925 22:44:54.199628 17748 ts_manager.cc:194] Registered new tserver with Master: 1ffb9543e6c340f78a7f82ffff34faf9 (127.10.198.3:46715)
I20260925 22:44:54.199890 18403 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 36158ad8c29246c2ab5424b4baf3038d [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 7 to 9, ed073dd0b5e2464bbc701bc2862011ea (127.10.198.60) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "36158ad8c29246c2ab5424b4baf3038d" member_type: VOTER last_known_addr { host: "127.10.198.62" port: 41969 } } peers { permanent_uuid: "f7d2a2767543443e8ba59b91bd97f4e1" member_type: VOTER last_known_addr { host: "127.10.198.61" port: 39135 } attrs { promote: false } } peers { permanent_uuid: "ed073dd0b5e2464bbc701bc2862011ea" member_type: VOTER last_known_addr { host: "127.10.198.60" port: 44389 } attrs { promote: false } } }
I20260925 22:44:54.200757 17758 raft_consensus.cc:2932] T 00000000000000000000000000000000 P f7d2a2767543443e8ba59b91bd97f4e1 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, ed073dd0b5e2464bbc701bc2862011ea (127.10.198.60) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "36158ad8c29246c2ab5424b4baf3038d" member_type: VOTER last_known_addr { host: "127.10.198.62" port: 41969 } } peers { permanent_uuid: "f7d2a2767543443e8ba59b91bd97f4e1" member_type: VOTER last_known_addr { host: "127.10.198.61" port: 39135 } attrs { promote: false } } peers { permanent_uuid: "ed073dd0b5e2464bbc701bc2862011ea" member_type: VOTER last_known_addr { host: "127.10.198.60" port: 44389 } attrs { promote: false } } }
I20260925 22:44:54.200870 17941 raft_consensus.cc:2932] T 00000000000000000000000000000000 P ed073dd0b5e2464bbc701bc2862011ea [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, ed073dd0b5e2464bbc701bc2862011ea (127.10.198.60) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "36158ad8c29246c2ab5424b4baf3038d" member_type: VOTER last_known_addr { host: "127.10.198.62" port: 41969 } } peers { permanent_uuid: "f7d2a2767543443e8ba59b91bd97f4e1" member_type: VOTER last_known_addr { host: "127.10.198.61" port: 39135 } attrs { promote: false } } peers { permanent_uuid: "ed073dd0b5e2464bbc701bc2862011ea" member_type: VOTER last_known_addr { host: "127.10.198.60" port: 44389 } attrs { promote: false } } }
I20260925 22:44:54.201853 17894 sys_catalog.cc:455] T 00000000000000000000000000000000 P 36158ad8c29246c2ab5424b4baf3038d [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "36158ad8c29246c2ab5424b4baf3038d" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "36158ad8c29246c2ab5424b4baf3038d" member_type: VOTER last_known_addr { host: "127.10.198.62" port: 41969 } } peers { permanent_uuid: "f7d2a2767543443e8ba59b91bd97f4e1" member_type: VOTER last_known_addr { host: "127.10.198.61" port: 39135 } attrs { promote: false } } peers { permanent_uuid: "ed073dd0b5e2464bbc701bc2862011ea" member_type: VOTER last_known_addr { host: "127.10.198.60" port: 44389 } attrs { promote: false } } }
I20260925 22:44:54.201897 18403 sys_catalog.cc:455] T 00000000000000000000000000000000 P 36158ad8c29246c2ab5424b4baf3038d [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "36158ad8c29246c2ab5424b4baf3038d" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "36158ad8c29246c2ab5424b4baf3038d" member_type: VOTER last_known_addr { host: "127.10.198.62" port: 41969 } } peers { permanent_uuid: "f7d2a2767543443e8ba59b91bd97f4e1" member_type: VOTER last_known_addr { host: "127.10.198.61" port: 39135 } attrs { promote: false } } peers { permanent_uuid: "ed073dd0b5e2464bbc701bc2862011ea" member_type: VOTER last_known_addr { host: "127.10.198.60" port: 44389 } attrs { promote: false } } }
I20260925 22:44:54.201996 17894 sys_catalog.cc:458] T 00000000000000000000000000000000 P 36158ad8c29246c2ab5424b4baf3038d [sys.catalog]: This master's current role is: LEADER
I20260925 22:44:54.202150 17894 sys_catalog.cc:455] T 00000000000000000000000000000000 P 36158ad8c29246c2ab5424b4baf3038d [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "36158ad8c29246c2ab5424b4baf3038d" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "36158ad8c29246c2ab5424b4baf3038d" member_type: VOTER last_known_addr { host: "127.10.198.62" port: 41969 } } peers { permanent_uuid: "f7d2a2767543443e8ba59b91bd97f4e1" member_type: VOTER last_known_addr { host: "127.10.198.61" port: 39135 } attrs { promote: false } } peers { permanent_uuid: "ed073dd0b5e2464bbc701bc2862011ea" member_type: VOTER last_known_addr { host: "127.10.198.60" port: 44389 } attrs { promote: false } } }
I20260925 22:44:54.202237 17894 sys_catalog.cc:458] T 00000000000000000000000000000000 P 36158ad8c29246c2ab5424b4baf3038d [sys.catalog]: This master's current role is: LEADER
I20260925 22:44:54.202004 18403 sys_catalog.cc:458] T 00000000000000000000000000000000 P 36158ad8c29246c2ab5424b4baf3038d [sys.catalog]: This master's current role is: LEADER
I20260925 22:44:54.203217 17973 sys_catalog.cc:455] T 00000000000000000000000000000000 P ed073dd0b5e2464bbc701bc2862011ea [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "36158ad8c29246c2ab5424b4baf3038d" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "36158ad8c29246c2ab5424b4baf3038d" member_type: VOTER last_known_addr { host: "127.10.198.62" port: 41969 } } peers { permanent_uuid: "f7d2a2767543443e8ba59b91bd97f4e1" member_type: VOTER last_known_addr { host: "127.10.198.61" port: 39135 } attrs { promote: false } } peers { permanent_uuid: "ed073dd0b5e2464bbc701bc2862011ea" member_type: VOTER last_known_addr { host: "127.10.198.60" port: 44389 } attrs { promote: false } } }
I20260925 22:44:54.203349 17973 sys_catalog.cc:458] T 00000000000000000000000000000000 P ed073dd0b5e2464bbc701bc2862011ea [sys.catalog]: This master's current role is: FOLLOWER
I20260925 22:44:54.206923 18411 sys_catalog.cc:455] T 00000000000000000000000000000000 P f7d2a2767543443e8ba59b91bd97f4e1 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "36158ad8c29246c2ab5424b4baf3038d" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "36158ad8c29246c2ab5424b4baf3038d" member_type: VOTER last_known_addr { host: "127.10.198.62" port: 41969 } } peers { permanent_uuid: "f7d2a2767543443e8ba59b91bd97f4e1" member_type: VOTER last_known_addr { host: "127.10.198.61" port: 39135 } attrs { promote: false } } peers { permanent_uuid: "ed073dd0b5e2464bbc701bc2862011ea" member_type: VOTER last_known_addr { host: "127.10.198.60" port: 44389 } attrs { promote: false } } }
I20260925 22:44:54.207127 18411 sys_catalog.cc:458] T 00000000000000000000000000000000 P f7d2a2767543443e8ba59b91bd97f4e1 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 22:44:54.212405 18255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:44:54.205312 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.198.2:35073 (local address 127.0.0.1:42508)
0925 22:44:54.205956 (+ 644us) negotiation.cc:107] Waiting for socket to connect
0925 22:44:54.205974 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0925 22:44:54.206078 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:44:54.208075 (+ 1997us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:44:54.208079 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:44:54.208101 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:44:54.208197 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:44:54.208203 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:44:54.209578 (+ 1375us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:44:54.209582 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:44:54.210699 (+ 1117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:44:54.210711 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:44:54.211890 (+ 1179us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:44:54.211893 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:44:54.212040 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:44:54.212068 (+ 28us) client_negotiation.cc:769] Sending connection context
0925 22:44:54.212140 (+ 72us) client_negotiation.cc:240] Negotiation successful
0925 22:44:54.212240 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":545,"mutex_wait_us":38}
I20260925 22:44:54.212581 18412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:44:54.205479 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42508 (local address 127.10.198.2:35073)
0925 22:44:54.207453 (+ 1974us) server_negotiation.cc:206] Beginning negotiation
0925 22:44:54.207469 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0925 22:44:54.207497 (+ 28us) server_negotiation.cc:407] Connection header received
0925 22:44:54.207587 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:44:54.207595 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:44:54.207811 (+ 216us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:44:54.208098 (+ 287us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:44:54.208294 (+ 196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:44:54.209426 (+ 1132us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:44:54.210855 (+ 1429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:44:54.211736 (+ 881us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:44:54.212074 (+ 338us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:44:54.212252 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0925 22:44:54.212313 (+ 61us) server_negotiation.cc:299] Negotiation successful
0925 22:44:54.212417 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1856,"thread_start_us":144,"threads_started":1}
I20260925 22:44:54.213052 18415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:44:54.205611 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.198.3:46715 (local address 127.0.0.1:44688)
0925 22:44:54.206531 (+ 920us) negotiation.cc:107] Waiting for socket to connect
0925 22:44:54.206549 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0925 22:44:54.206655 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:44:54.207321 (+ 666us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:44:54.207326 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:44:54.207350 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:44:54.207491 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:44:54.207499 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:44:54.208877 (+ 1378us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:44:54.208880 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:44:54.209945 (+ 1065us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:44:54.209957 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:44:54.211810 (+ 1853us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:44:54.211816 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:44:54.212724 (+ 908us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:44:54.212754 (+ 30us) client_negotiation.cc:769] Sending connection context
0925 22:44:54.212829 (+ 75us) client_negotiation.cc:240] Negotiation successful
0925 22:44:54.212911 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":852,"mutex_wait_us":39,"thread_start_us":118,"threads_started":1}
I20260925 22:44:54.213413 18413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:44:54.205978 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.198.1:41135 (local address 127.0.0.1:48082)
0925 22:44:54.208058 (+ 2080us) negotiation.cc:107] Waiting for socket to connect
0925 22:44:54.208075 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 22:44:54.208176 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:44:54.208814 (+ 638us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:44:54.208817 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:44:54.208841 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:44:54.208939 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:44:54.208945 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:44:54.210312 (+ 1367us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:44:54.210315 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:44:54.211581 (+ 1266us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:44:54.211594 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:44:54.212934 (+ 1340us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:44:54.212937 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:44:54.213016 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:44:54.213036 (+ 20us) client_negotiation.cc:769] Sending connection context
0925 22:44:54.213150 (+ 114us) client_negotiation.cc:240] Negotiation successful
0925 22:44:54.213246 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1996,"mutex_wait_us":46,"thread_start_us":112,"threads_started":1}
I20260925 22:44:54.213868 18416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:44:54.206162 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48082 (local address 127.10.198.1:41135)
0925 22:44:54.206449 (+ 287us) server_negotiation.cc:206] Beginning negotiation
0925 22:44:54.206459 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 22:44:54.208302 (+ 1843us) server_negotiation.cc:407] Connection header received
0925 22:44:54.208399 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:44:54.208405 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:44:54.208673 (+ 268us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:44:54.208839 (+ 166us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:44:54.209038 (+ 199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:44:54.209958 (+ 920us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:44:54.211752 (+ 1794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:44:54.212552 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:44:54.212700 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:44:54.212850 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0925 22:44:54.213355 (+ 505us) server_negotiation.cc:299] Negotiation successful
0925 22:44:54.213745 (+ 390us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":176,"thread_start_us":94,"threads_started":1}
I20260925 22:44:54.216048 18181 tablet_service.cc:1467] Processing CreateTablet for tablet 6e7a24f5c47d420a851c32de00082a1f (DEFAULT_TABLE table=test-table [id=5e2d980ef4e246beac5d85c119369c87]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 22:44:54.217345 18181 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6e7a24f5c47d420a851c32de00082a1f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:44:54.217139 18044 tablet_service.cc:1467] Processing CreateTablet for tablet 6e7a24f5c47d420a851c32de00082a1f (DEFAULT_TABLE table=test-table [id=5e2d980ef4e246beac5d85c119369c87]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 22:44:54.218247 18044 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6e7a24f5c47d420a851c32de00082a1f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:44:54.224100 18414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:44:54.205790 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44688 (local address 127.10.198.3:46715)
0925 22:44:54.206587 (+ 797us) server_negotiation.cc:206] Beginning negotiation
0925 22:44:54.206595 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 22:44:54.206618 (+ 23us) server_negotiation.cc:407] Connection header received
0925 22:44:54.206813 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:44:54.206823 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:44:54.207108 (+ 285us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:44:54.207345 (+ 237us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:44:54.207612 (+ 267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:44:54.208744 (+ 1132us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:44:54.210710 (+ 1966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:44:54.211607 (+ 897us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:44:54.211815 (+ 208us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:44:54.212047 (+ 232us) server_negotiation.cc:1035] Waiting for connection context
0925 22:44:54.213346 (+ 1299us) server_negotiation.cc:299] Negotiation successful
0925 22:44:54.213469 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":702,"thread_start_us":130,"threads_started":1}
I20260925 22:44:54.227643 18418 tablet_bootstrap.cc:492] T 6e7a24f5c47d420a851c32de00082a1f P f5cc83f491e5409895f3eaf63bdc05c6: Bootstrap starting.
I20260925 22:44:54.227984 18417 tablet_bootstrap.cc:492] T 6e7a24f5c47d420a851c32de00082a1f P dcfd1608a17742d9a03fa549b12bb0c8: Bootstrap starting.
I20260925 22:44:54.227596 18318 tablet_service.cc:1467] Processing CreateTablet for tablet 6e7a24f5c47d420a851c32de00082a1f (DEFAULT_TABLE table=test-table [id=5e2d980ef4e246beac5d85c119369c87]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 22:44:54.228890 18318 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6e7a24f5c47d420a851c32de00082a1f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:44:54.230484 18418 tablet_bootstrap.cc:654] T 6e7a24f5c47d420a851c32de00082a1f P f5cc83f491e5409895f3eaf63bdc05c6: Neither blocks nor log segments found. Creating new log.
I20260925 22:44:54.230484 18417 tablet_bootstrap.cc:654] T 6e7a24f5c47d420a851c32de00082a1f P dcfd1608a17742d9a03fa549b12bb0c8: Neither blocks nor log segments found. Creating new log.
I20260925 22:44:54.231508 18417 log.cc:824] T 6e7a24f5c47d420a851c32de00082a1f P dcfd1608a17742d9a03fa549b12bb0c8: Log is configured to *not* fsync() on all Append() calls
I20260925 22:44:54.231508 18418 log.cc:824] T 6e7a24f5c47d420a851c32de00082a1f P f5cc83f491e5409895f3eaf63bdc05c6: Log is configured to *not* fsync() on all Append() calls
I20260925 22:44:54.233942 18417 tablet_bootstrap.cc:492] T 6e7a24f5c47d420a851c32de00082a1f P dcfd1608a17742d9a03fa549b12bb0c8: No bootstrap required, opened a new log
I20260925 22:44:54.233940 18418 tablet_bootstrap.cc:492] T 6e7a24f5c47d420a851c32de00082a1f P f5cc83f491e5409895f3eaf63bdc05c6: No bootstrap required, opened a new log
I20260925 22:44:54.234257 18418 ts_tablet_manager.cc:1397] T 6e7a24f5c47d420a851c32de00082a1f P f5cc83f491e5409895f3eaf63bdc05c6: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20260925 22:44:54.234287 18417 ts_tablet_manager.cc:1397] T 6e7a24f5c47d420a851c32de00082a1f P dcfd1608a17742d9a03fa549b12bb0c8: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20260925 22:44:54.238016 18421 tablet_bootstrap.cc:492] T 6e7a24f5c47d420a851c32de00082a1f P 1ffb9543e6c340f78a7f82ffff34faf9: Bootstrap starting.
I20260925 22:44:54.238683 18418 raft_consensus.cc:348] T 6e7a24f5c47d420a851c32de00082a1f P f5cc83f491e5409895f3eaf63bdc05c6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5cc83f491e5409895f3eaf63bdc05c6" member_type: VOTER last_known_addr { host: "127.10.198.2" port: 35073 } } peers { permanent_uuid: "1ffb9543e6c340f78a7f82ffff34faf9" member_type: VOTER last_known_addr { host: "127.10.198.3" port: 46715 } } peers { permanent_uuid: "dcfd1608a17742d9a03fa549b12bb0c8" member_type: VOTER last_known_addr { host: "127.10.198.1" port: 41135 } }
I20260925 22:44:54.238711 18417 raft_consensus.cc:348] T 6e7a24f5c47d420a851c32de00082a1f P dcfd1608a17742d9a03fa549b12bb0c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5cc83f491e5409895f3eaf63bdc05c6" member_type: VOTER last_known_addr { host: "127.10.198.2" port: 35073 } } peers { permanent_uuid: "1ffb9543e6c340f78a7f82ffff34faf9" member_type: VOTER last_known_addr { host: "127.10.198.3" port: 46715 } } peers { permanent_uuid: "dcfd1608a17742d9a03fa549b12bb0c8" member_type: VOTER last_known_addr { host: "127.10.198.1" port: 41135 } }
I20260925 22:44:54.239024 18418 raft_consensus.cc:374] T 6e7a24f5c47d420a851c32de00082a1f P f5cc83f491e5409895f3eaf63bdc05c6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:44:54.239044 18417 raft_consensus.cc:374] T 6e7a24f5c47d420a851c32de00082a1f P dcfd1608a17742d9a03fa549b12bb0c8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:44:54.239111 18417 raft_consensus.cc:729] T 6e7a24f5c47d420a851c32de00082a1f P dcfd1608a17742d9a03fa549b12bb0c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dcfd1608a17742d9a03fa549b12bb0c8, State: Initialized, Role: FOLLOWER
I20260925 22:44:54.239102 18418 raft_consensus.cc:729] T 6e7a24f5c47d420a851c32de00082a1f P f5cc83f491e5409895f3eaf63bdc05c6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f5cc83f491e5409895f3eaf63bdc05c6, State: Initialized, Role: FOLLOWER
I20260925 22:44:54.239632 18418 consensus_queue.cc:260] T 6e7a24f5c47d420a851c32de00082a1f P f5cc83f491e5409895f3eaf63bdc05c6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5cc83f491e5409895f3eaf63bdc05c6" member_type: VOTER last_known_addr { host: "127.10.198.2" port: 35073 } } peers { permanent_uuid: "1ffb9543e6c340f78a7f82ffff34faf9" member_type: VOTER last_known_addr { host: "127.10.198.3" port: 46715 } } peers { permanent_uuid: "dcfd1608a17742d9a03fa549b12bb0c8" member_type: VOTER last_known_addr { host: "127.10.198.1" port: 41135 } }
I20260925 22:44:54.239629 18417 consensus_queue.cc:260] T 6e7a24f5c47d420a851c32de00082a1f P dcfd1608a17742d9a03fa549b12bb0c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5cc83f491e5409895f3eaf63bdc05c6" member_type: VOTER last_known_addr { host: "127.10.198.2" port: 35073 } } peers { permanent_uuid: "1ffb9543e6c340f78a7f82ffff34faf9" member_type: VOTER last_known_addr { host: "127.10.198.3" port: 46715 } } peers { permanent_uuid: "dcfd1608a17742d9a03fa549b12bb0c8" member_type: VOTER last_known_addr { host: "127.10.198.1" port: 41135 } }
I20260925 22:44:54.240808 18417 ts_tablet_manager.cc:1428] T 6e7a24f5c47d420a851c32de00082a1f P dcfd1608a17742d9a03fa549b12bb0c8: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260925 22:44:54.240921 18418 ts_tablet_manager.cc:1428] T 6e7a24f5c47d420a851c32de00082a1f P f5cc83f491e5409895f3eaf63bdc05c6: Time spent starting tablet: real 0.007s user 0.002s sys 0.004s
I20260925 22:44:54.240998 18111 heartbeater.cc:503] Master 127.10.198.62:41969 was elected leader, sending a full tablet report...
I20260925 22:44:54.241045 18421 tablet_bootstrap.cc:654] T 6e7a24f5c47d420a851c32de00082a1f P 1ffb9543e6c340f78a7f82ffff34faf9: Neither blocks nor log segments found. Creating new log.
I20260925 22:44:54.241286 18248 heartbeater.cc:503] Master 127.10.198.62:41969 was elected leader, sending a full tablet report...
I20260925 22:44:54.243566 18421 log.cc:824] T 6e7a24f5c47d420a851c32de00082a1f P 1ffb9543e6c340f78a7f82ffff34faf9: Log is configured to *not* fsync() on all Append() calls
I20260925 22:44:54.246563 18421 tablet_bootstrap.cc:492] T 6e7a24f5c47d420a851c32de00082a1f P 1ffb9543e6c340f78a7f82ffff34faf9: No bootstrap required, opened a new log
I20260925 22:44:54.246822 18421 ts_tablet_manager.cc:1397] T 6e7a24f5c47d420a851c32de00082a1f P 1ffb9543e6c340f78a7f82ffff34faf9: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260925 22:44:54.250881 18421 raft_consensus.cc:348] T 6e7a24f5c47d420a851c32de00082a1f P 1ffb9543e6c340f78a7f82ffff34faf9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5cc83f491e5409895f3eaf63bdc05c6" member_type: VOTER last_known_addr { host: "127.10.198.2" port: 35073 } } peers { permanent_uuid: "1ffb9543e6c340f78a7f82ffff34faf9" member_type: VOTER last_known_addr { host: "127.10.198.3" port: 46715 } } peers { permanent_uuid: "dcfd1608a17742d9a03fa549b12bb0c8" member_type: VOTER last_known_addr { host: "127.10.198.1" port: 41135 } }
I20260925 22:44:54.251169 18421 raft_consensus.cc:374] T 6e7a24f5c47d420a851c32de00082a1f P 1ffb9543e6c340f78a7f82ffff34faf9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:44:54.251214 18421 raft_consensus.cc:729] T 6e7a24f5c47d420a851c32de00082a1f P 1ffb9543e6c340f78a7f82ffff34faf9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1ffb9543e6c340f78a7f82ffff34faf9, State: Initialized, Role: FOLLOWER
I20260925 22:44:54.251643 18421 consensus_queue.cc:260] T 6e7a24f5c47d420a851c32de00082a1f P 1ffb9543e6c340f78a7f82ffff34faf9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5cc83f491e5409895f3eaf63bdc05c6" member_type: VOTER last_known_addr { host: "127.10.198.2" port: 35073 } } peers { permanent_uuid: "1ffb9543e6c340f78a7f82ffff34faf9" member_type: VOTER last_known_addr { host: "127.10.198.3" port: 46715 } } peers { permanent_uuid: "dcfd1608a17742d9a03fa549b12bb0c8" member_type: VOTER last_known_addr { host: "127.10.198.1" port: 41135 } }
I20260925 22:44:54.252486 18385 heartbeater.cc:503] Master 127.10.198.62:41969 was elected leader, sending a full tablet report...
I20260925 22:44:54.252908 18421 ts_tablet_manager.cc:1428] T 6e7a24f5c47d420a851c32de00082a1f P 1ffb9543e6c340f78a7f82ffff34faf9: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260925 22:44:54.305032 18422 raft_consensus.cc:482] T 6e7a24f5c47d420a851c32de00082a1f P dcfd1608a17742d9a03fa549b12bb0c8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 22:44:54.305413 18422 raft_consensus.cc:504] T 6e7a24f5c47d420a851c32de00082a1f P dcfd1608a17742d9a03fa549b12bb0c8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5cc83f491e5409895f3eaf63bdc05c6" member_type: VOTER last_known_addr { host: "127.10.198.2" port: 35073 } } peers { permanent_uuid: "1ffb9543e6c340f78a7f82ffff34faf9" member_type: VOTER last_known_addr { host: "127.10.198.3" port: 46715 } } peers { permanent_uuid: "dcfd1608a17742d9a03fa549b12bb0c8" member_type: VOTER last_known_addr { host: "127.10.198.1" port: 41135 } }
I20260925 22:44:54.306823 18422 leader_election.cc:288] T 6e7a24f5c47d420a851c32de00082a1f P dcfd1608a17742d9a03fa549b12bb0c8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f5cc83f491e5409895f3eaf63bdc05c6 (127.10.198.2:35073), 1ffb9543e6c340f78a7f82ffff34faf9 (127.10.198.3:46715)
I20260925 22:44:54.310984 18426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:44:54.307041 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.198.2:35073 (local address 127.10.198.1:50445)
0925 22:44:54.307359 (+ 318us) negotiation.cc:107] Waiting for socket to connect
0925 22:44:54.307376 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 22:44:54.307496 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:44:54.307973 (+ 477us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:44:54.307975 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:44:54.307992 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:44:54.308098 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:44:54.308103 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:44:54.308881 (+ 778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:44:54.308887 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:44:54.309751 (+ 864us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:44:54.309760 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:44:54.310464 (+ 704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:44:54.310467 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:44:54.310587 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:44:54.310608 (+ 21us) client_negotiation.cc:769] Sending connection context
0925 22:44:54.310708 (+ 100us) client_negotiation.cc:240] Negotiation successful
0925 22:44:54.310802 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":233,"mutex_wait_us":1,"thread_start_us":146,"threads_started":1}
I20260925 22:44:54.310951 18412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:44:54.307200 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.198.1:50445 (local address 127.10.198.2:35073)
0925 22:44:54.307369 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0925 22:44:54.307373 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:44:54.307464 (+ 91us) server_negotiation.cc:407] Connection header received
0925 22:44:54.307664 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:44:54.307668 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:44:54.307735 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:44:54.307779 (+ 44us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:44:54.308230 (+ 451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:44:54.308761 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:44:54.309895 (+ 1134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:44:54.310310 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:44:54.310458 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:44:54.310519 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0925 22:44:54.310725 (+ 206us) server_negotiation.cc:299] Negotiation successful
0925 22:44:54.310804 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260925 22:44:54.311239 18414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:44:54.307631 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.198.1:56799 (local address 127.10.198.3:46715)
0925 22:44:54.307765 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0925 22:44:54.307771 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 22:44:54.307833 (+ 62us) server_negotiation.cc:407] Connection header received
0925 22:44:54.308051 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:44:54.308055 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:44:54.308110 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:44:54.308207 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:44:54.308484 (+ 277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:44:54.308999 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:44:54.310101 (+ 1102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:44:54.310551 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:44:54.310699 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:44:54.310761 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0925 22:44:54.310976 (+ 215us) server_negotiation.cc:299] Negotiation successful
0925 22:44:54.311060 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260925 22:44:54.311453 18427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:44:54.307487 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.198.3:46715 (local address 127.10.198.1:56799)
0925 22:44:54.307734 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0925 22:44:54.307750 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 22:44:54.307914 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:44:54.308217 (+ 303us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:44:54.308220 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:44:54.308236 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:44:54.308359 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:44:54.308366 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:44:54.309126 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:44:54.309132 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:44:54.309951 (+ 819us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:44:54.309960 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:44:54.310699 (+ 739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:44:54.310703 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:44:54.310832 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:44:54.310864 (+ 32us) client_negotiation.cc:769] Sending connection context
0925 22:44:54.310961 (+ 97us) client_negotiation.cc:240] Negotiation successful
0925 22:44:54.311209 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":174,"mutex_wait_us":22,"thread_start_us":95,"threads_started":1}
I20260925 22:44:54.311699 18201 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e7a24f5c47d420a851c32de00082a1f" candidate_uuid: "dcfd1608a17742d9a03fa549b12bb0c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f5cc83f491e5409895f3eaf63bdc05c6" is_pre_election: true
I20260925 22:44:54.312080 18201 raft_consensus.cc:2454] T 6e7a24f5c47d420a851c32de00082a1f P f5cc83f491e5409895f3eaf63bdc05c6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dcfd1608a17742d9a03fa549b12bb0c8 in term 0.
I20260925 22:44:54.312088 18338 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e7a24f5c47d420a851c32de00082a1f" candidate_uuid: "dcfd1608a17742d9a03fa549b12bb0c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1ffb9543e6c340f78a7f82ffff34faf9" is_pre_election: true
I20260925 22:44:54.312338 18338 raft_consensus.cc:2454] T 6e7a24f5c47d420a851c32de00082a1f P 1ffb9543e6c340f78a7f82ffff34faf9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dcfd1608a17742d9a03fa549b12bb0c8 in term 0.
I20260925 22:44:54.312666 17998 leader_election.cc:302] T 6e7a24f5c47d420a851c32de00082a1f P dcfd1608a17742d9a03fa549b12bb0c8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: dcfd1608a17742d9a03fa549b12bb0c8, f5cc83f491e5409895f3eaf63bdc05c6; no voters:
I20260925 22:44:54.313001 18422 raft_consensus.cc:2781] T 6e7a24f5c47d420a851c32de00082a1f P dcfd1608a17742d9a03fa549b12bb0c8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 22:44:54.313083 18422 raft_consensus.cc:482] T 6e7a24f5c47d420a851c32de00082a1f P dcfd1608a17742d9a03fa549b12bb0c8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 22:44:54.313115 18422 raft_consensus.cc:3037] T 6e7a24f5c47d420a851c32de00082a1f P dcfd1608a17742d9a03fa549b12bb0c8 [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:44:54.314330 18422 raft_consensus.cc:504] T 6e7a24f5c47d420a851c32de00082a1f P dcfd1608a17742d9a03fa549b12bb0c8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5cc83f491e5409895f3eaf63bdc05c6" member_type: VOTER last_known_addr { host: "127.10.198.2" port: 35073 } } peers { permanent_uuid: "1ffb9543e6c340f78a7f82ffff34faf9" member_type: VOTER last_known_addr { host: "127.10.198.3" port: 46715 } } peers { permanent_uuid: "dcfd1608a17742d9a03fa549b12bb0c8" member_type: VOTER last_known_addr { host: "127.10.198.1" port: 41135 } }
I20260925 22:44:54.314756 18422 leader_election.cc:288] T 6e7a24f5c47d420a851c32de00082a1f P dcfd1608a17742d9a03fa549b12bb0c8 [CANDIDATE]: Term 1 election: Requested vote from peers f5cc83f491e5409895f3eaf63bdc05c6 (127.10.198.2:35073), 1ffb9543e6c340f78a7f82ffff34faf9 (127.10.198.3:46715)
I20260925 22:44:54.315294 18201 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e7a24f5c47d420a851c32de00082a1f" candidate_uuid: "dcfd1608a17742d9a03fa549b12bb0c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f5cc83f491e5409895f3eaf63bdc05c6"
I20260925 22:44:54.315299 18338 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e7a24f5c47d420a851c32de00082a1f" candidate_uuid: "dcfd1608a17742d9a03fa549b12bb0c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1ffb9543e6c340f78a7f82ffff34faf9"
I20260925 22:44:54.315457 18201 raft_consensus.cc:3037] T 6e7a24f5c47d420a851c32de00082a1f P f5cc83f491e5409895f3eaf63bdc05c6 [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:44:54.315457 18338 raft_consensus.cc:3037] T 6e7a24f5c47d420a851c32de00082a1f P 1ffb9543e6c340f78a7f82ffff34faf9 [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:44:54.317039 18201 raft_consensus.cc:2454] T 6e7a24f5c47d420a851c32de00082a1f P f5cc83f491e5409895f3eaf63bdc05c6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dcfd1608a17742d9a03fa549b12bb0c8 in term 1.
I20260925 22:44:54.317039 18338 raft_consensus.cc:2454] T 6e7a24f5c47d420a851c32de00082a1f P 1ffb9543e6c340f78a7f82ffff34faf9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dcfd1608a17742d9a03fa549b12bb0c8 in term 1.
I20260925 22:44:54.317497 17998 leader_election.cc:302] T 6e7a24f5c47d420a851c32de00082a1f P dcfd1608a17742d9a03fa549b12bb0c8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1ffb9543e6c340f78a7f82ffff34faf9, dcfd1608a17742d9a03fa549b12bb0c8; no voters:
I20260925 22:44:54.317797 18422 raft_consensus.cc:2781] T 6e7a24f5c47d420a851c32de00082a1f P dcfd1608a17742d9a03fa549b12bb0c8 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 22:44:54.318063 18422 raft_consensus.cc:686] T 6e7a24f5c47d420a851c32de00082a1f P dcfd1608a17742d9a03fa549b12bb0c8 [term 1 LEADER]: Becoming Leader. State: Replica: dcfd1608a17742d9a03fa549b12bb0c8, State: Running, Role: LEADER
I20260925 22:44:54.318460 18422 consensus_queue.cc:237] T 6e7a24f5c47d420a851c32de00082a1f P dcfd1608a17742d9a03fa549b12bb0c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5cc83f491e5409895f3eaf63bdc05c6" member_type: VOTER last_known_addr { host: "127.10.198.2" port: 35073 } } peers { permanent_uuid: "1ffb9543e6c340f78a7f82ffff34faf9" member_type: VOTER last_known_addr { host: "127.10.198.3" port: 46715 } } peers { permanent_uuid: "dcfd1608a17742d9a03fa549b12bb0c8" member_type: VOTER last_known_addr { host: "127.10.198.1" port: 41135 } }
I20260925 22:44:54.323041 17147 catalog_manager.cc:5495] T 6e7a24f5c47d420a851c32de00082a1f P dcfd1608a17742d9a03fa549b12bb0c8 reported cstate change: term changed from 0 to 1, leader changed from <none> to dcfd1608a17742d9a03fa549b12bb0c8 (127.10.198.1). New cstate: current_term: 1 leader_uuid: "dcfd1608a17742d9a03fa549b12bb0c8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5cc83f491e5409895f3eaf63bdc05c6" member_type: VOTER last_known_addr { host: "127.10.198.2" port: 35073 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1ffb9543e6c340f78a7f82ffff34faf9" member_type: VOTER last_known_addr { host: "127.10.198.3" port: 46715 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dcfd1608a17742d9a03fa549b12bb0c8" member_type: VOTER last_known_addr { host: "127.10.198.1" port: 41135 } health_report { overall_health: HEALTHY } } }
I20260925 22:44:54.338281 18416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:44:54.334204 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48092 (local address 127.10.198.1:41135)
0925 22:44:54.334374 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0925 22:44:54.334379 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:44:54.334395 (+ 16us) server_negotiation.cc:407] Connection header received
0925 22:44:54.334562 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:44:54.334565 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:44:54.334635 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:44:54.334756 (+ 121us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 22:44:54.335049 (+ 293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:44:54.335632 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:44:54.336657 (+ 1025us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:44:54.336902 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:44:54.336999 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:44:54.337310 (+ 311us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 22:44:54.337784 (+ 474us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 22:44:54.337880 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0925 22:44:54.337990 (+ 110us) server_negotiation.cc:299] Negotiation successful
0925 22:44:54.338052 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260925 22:44:54.345072 18201 raft_consensus.cc:1261] T 6e7a24f5c47d420a851c32de00082a1f P f5cc83f491e5409895f3eaf63bdc05c6 [term 1 FOLLOWER]: Refusing update from remote peer dcfd1608a17742d9a03fa549b12bb0c8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 22:44:54.345162 18338 raft_consensus.cc:1261] T 6e7a24f5c47d420a851c32de00082a1f P 1ffb9543e6c340f78a7f82ffff34faf9 [term 1 FOLLOWER]: Refusing update from remote peer dcfd1608a17742d9a03fa549b12bb0c8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 22:44:54.345939 18428 consensus_queue.cc:1035] T 6e7a24f5c47d420a851c32de00082a1f P dcfd1608a17742d9a03fa549b12bb0c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f5cc83f491e5409895f3eaf63bdc05c6" member_type: VOTER last_known_addr { host: "127.10.198.2" port: 35073 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 22:44:54.346164 18422 consensus_queue.cc:1035] T 6e7a24f5c47d420a851c32de00082a1f P dcfd1608a17742d9a03fa549b12bb0c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1ffb9543e6c340f78a7f82ffff34faf9" member_type: VOTER last_known_addr { host: "127.10.198.3" port: 46715 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 22:44:54.354115 18431 mvcc.cc:205] Tried to move back new op lower bound from 7333381301628616704 to 7333381301531435008. Current Snapshot: MvccSnapshot[applied={T|T < 7333381301628616704}]
I20260925 22:44:54.355957 18432 mvcc.cc:205] Tried to move back new op lower bound from 7333381301628616704 to 7333381301531435008. Current Snapshot: MvccSnapshot[applied={T|T < 7333381301628616704}]
I20260925 22:44:54.368358 17145 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42110:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 22:44:54.368620 17145 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42110:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 22:44:54.381237 17145 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 36158ad8c29246c2ab5424b4baf3038d: Sending DeleteTablet for 3 replicas of tablet 6e7a24f5c47d420a851c32de00082a1f
I20260925 22:44:54.382308 18318 tablet_service.cc:1514] Processing DeleteTablet for tablet 6e7a24f5c47d420a851c32de00082a1f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 22:44:54 UTC) from {username='oryx', principal='oryx/127.10.198.62@KRBTEST.COM'} at 127.0.0.1:44688
I20260925 22:44:54.382592 18317 tablet_service.cc:1514] Processing DeleteTablet for tablet 6e7a24f5c47d420a851c32de00082a1f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 22:44:54 UTC) from {username='oryx', principal='oryx/127.10.198.62@KRBTEST.COM'} at 127.0.0.1:44688
I20260925 22:44:54.382959 18181 tablet_service.cc:1514] Processing DeleteTablet for tablet 6e7a24f5c47d420a851c32de00082a1f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 22:44:54 UTC) from {username='oryx', principal='oryx/127.10.198.62@KRBTEST.COM'} at 127.0.0.1:42508
I20260925 22:44:54.383880 18044 tablet_service.cc:1514] Processing DeleteTablet for tablet 6e7a24f5c47d420a851c32de00082a1f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 22:44:54 UTC) from {username='oryx', principal='oryx/127.10.198.62@KRBTEST.COM'} at 127.0.0.1:48082
I20260925 22:44:54.384917 11032 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrWXm8X/build/debug/bin/kudu with pid 17115
I20260925 22:44:54.385040 18044 tablet_service.cc:1514] Processing DeleteTablet for tablet 6e7a24f5c47d420a851c32de00082a1f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 22:44:54 UTC) from {username='oryx', principal='oryx/127.10.198.62@KRBTEST.COM'} at 127.0.0.1:48082
I20260925 22:44:54.386162 18441 tablet_replica.cc:330] T 6e7a24f5c47d420a851c32de00082a1f P 1ffb9543e6c340f78a7f82ffff34faf9: stopping tablet replica
I20260925 22:44:54.386691 18441 raft_consensus.cc:2229] T 6e7a24f5c47d420a851c32de00082a1f P 1ffb9543e6c340f78a7f82ffff34faf9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 22:44:54.387204 18441 raft_consensus.cc:2258] T 6e7a24f5c47d420a851c32de00082a1f P 1ffb9543e6c340f78a7f82ffff34faf9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 22:44:54.387755 18442 tablet_replica.cc:330] T 6e7a24f5c47d420a851c32de00082a1f P f5cc83f491e5409895f3eaf63bdc05c6: stopping tablet replica
I20260925 22:44:54.388195 18442 raft_consensus.cc:2229] T 6e7a24f5c47d420a851c32de00082a1f P f5cc83f491e5409895f3eaf63bdc05c6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 22:44:54.388602 18442 raft_consensus.cc:2258] T 6e7a24f5c47d420a851c32de00082a1f P f5cc83f491e5409895f3eaf63bdc05c6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 22:44:54.389447 18441 ts_tablet_manager.cc:1905] T 6e7a24f5c47d420a851c32de00082a1f P 1ffb9543e6c340f78a7f82ffff34faf9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 22:44:54.391296 18442 ts_tablet_manager.cc:1905] T 6e7a24f5c47d420a851c32de00082a1f P f5cc83f491e5409895f3eaf63bdc05c6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 22:44:54.392916 18441 ts_tablet_manager.cc:1918] T 6e7a24f5c47d420a851c32de00082a1f P 1ffb9543e6c340f78a7f82ffff34faf9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 22:44:54.393157 18441 log.cc:1196] T 6e7a24f5c47d420a851c32de00082a1f P 1ffb9543e6c340f78a7f82ffff34faf9: Deleting WAL directory at /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376259650642-11032-0/minicluster-data/ts-2/wal/wals/6e7a24f5c47d420a851c32de00082a1f
I20260925 22:44:54.393599 18441 ts_tablet_manager.cc:1939] T 6e7a24f5c47d420a851c32de00082a1f P 1ffb9543e6c340f78a7f82ffff34faf9: Deleting consensus metadata
W20260925 22:44:54.395040 18271 connection.cc:441] server connection from 127.0.0.1:44688 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44688 (request call id 1) could send its response
I20260925 22:44:54.395115 18442 ts_tablet_manager.cc:1918] T 6e7a24f5c47d420a851c32de00082a1f P f5cc83f491e5409895f3eaf63bdc05c6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 22:44:54.395233 18442 log.cc:1196] T 6e7a24f5c47d420a851c32de00082a1f P f5cc83f491e5409895f3eaf63bdc05c6: Deleting WAL directory at /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376259650642-11032-0/minicluster-data/ts-1/wal/wals/6e7a24f5c47d420a851c32de00082a1f
I20260925 22:44:54.395632 18442 ts_tablet_manager.cc:1939] T 6e7a24f5c47d420a851c32de00082a1f P f5cc83f491e5409895f3eaf63bdc05c6: Deleting consensus metadata
W20260925 22:44:54.395776 18271 connection.cc:441] server connection from 127.0.0.1:44688 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44688 (request call id 2) could send its response
I20260925 22:44:54.397588 18444 tablet_replica.cc:330] T 6e7a24f5c47d420a851c32de00082a1f P dcfd1608a17742d9a03fa549b12bb0c8: stopping tablet replica
I20260925 22:44:54.398007 18444 raft_consensus.cc:2229] T 6e7a24f5c47d420a851c32de00082a1f P dcfd1608a17742d9a03fa549b12bb0c8 [term 1 LEADER]: Raft consensus shutting down.
I20260925 22:44:54.398459 18444 raft_consensus.cc:2258] T 6e7a24f5c47d420a851c32de00082a1f P dcfd1608a17742d9a03fa549b12bb0c8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 22:44:54.400647 18444 ts_tablet_manager.cc:1905] T 6e7a24f5c47d420a851c32de00082a1f P dcfd1608a17742d9a03fa549b12bb0c8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 22:44:54.404407 18444 ts_tablet_manager.cc:1918] T 6e7a24f5c47d420a851c32de00082a1f P dcfd1608a17742d9a03fa549b12bb0c8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 22:44:54.404549 18444 log.cc:1196] T 6e7a24f5c47d420a851c32de00082a1f P dcfd1608a17742d9a03fa549b12bb0c8: Deleting WAL directory at /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376259650642-11032-0/minicluster-data/ts-0/wal/wals/6e7a24f5c47d420a851c32de00082a1f
I20260925 22:44:54.404987 18444 ts_tablet_manager.cc:1939] T 6e7a24f5c47d420a851c32de00082a1f P dcfd1608a17742d9a03fa549b12bb0c8: Deleting consensus metadata
W20260925 22:44:54.406481 17998 connection.cc:441] server connection from 127.0.0.1:48082 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:48082 (request call id 1) could send its response
W20260925 22:44:54.406833 17998 connection.cc:441] server connection from 127.0.0.1:48082 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:48082 (request call id 2) could send its response
W20260925 22:44:54.409941 18135 connection.cc:441] server connection from 127.0.0.1:42508 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42508 (request call id 1) could send its response
I20260925 22:44:54.410956 17870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:44:54.384047 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59306 (local address 127.10.198.61:39135)
0925 22:44:54.384215 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0925 22:44:54.384220 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:44:54.389274 (+ 5054us) server_negotiation.cc:407] Connection header received
0925 22:44:54.389794 (+ 520us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:44:54.389798 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:44:54.389868 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:44:54.389937 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:44:54.397383 (+ 7446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:44:54.398132 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:44:54.399476 (+ 1344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:44:54.399950 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:44:54.400070 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:44:54.403035 (+ 2965us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:44:54.403039 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:44:54.403043 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:44:54.403083 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:44:54.408390 (+ 5307us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:44:54.408976 (+ 586us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:44:54.408980 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:44:54.408982 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:44:54.409094 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:44:54.409455 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:44:54.409459 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:44:54.409461 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:44:54.409699 (+ 238us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:44:54.409824 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0925 22:44:54.410574 (+ 750us) server_negotiation.cc:299] Negotiation successful
0925 22:44:54.410730 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260925 22:44:54.417495 18398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:44:54.404415 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45668 (local address 127.10.198.60:44389)
0925 22:44:54.404579 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0925 22:44:54.404586 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 22:44:54.405072 (+ 486us) server_negotiation.cc:407] Connection header received
0925 22:44:54.405404 (+ 332us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:44:54.405408 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:44:54.405470 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:44:54.405533 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:44:54.406704 (+ 1171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:44:54.407490 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:44:54.409305 (+ 1815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:44:54.409676 (+ 371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:44:54.409791 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:44:54.412681 (+ 2890us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:44:54.412687 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:44:54.412691 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:44:54.412720 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:44:54.415119 (+ 2399us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:44:54.415751 (+ 632us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:44:54.415755 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:44:54.415770 (+ 15us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:44:54.415907 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:44:54.416272 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:44:54.416277 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:44:54.416279 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:44:54.416505 (+ 226us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:44:54.416646 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0925 22:44:54.417169 (+ 523us) server_negotiation.cc:299] Negotiation successful
0925 22:44:54.417317 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260925 22:44:54.420946 18399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:44:54.407668 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59322 (local address 127.10.198.61:39135)
0925 22:44:54.407815 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0925 22:44:54.407820 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:44:54.407997 (+ 177us) server_negotiation.cc:407] Connection header received
0925 22:44:54.408052 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:44:54.408059 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:44:54.408120 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:44:54.408203 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:44:54.410916 (+ 2713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:44:54.411775 (+ 859us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:44:54.413287 (+ 1512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:44:54.413665 (+ 378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:44:54.413753 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:44:54.415916 (+ 2163us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:44:54.415921 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:44:54.415924 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:44:54.415954 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:44:54.418756 (+ 2802us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:44:54.419291 (+ 535us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:44:54.419295 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:44:54.419297 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:44:54.419391 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:44:54.419746 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:44:54.419749 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:44:54.419751 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:44:54.420062 (+ 311us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:44:54.420183 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0925 22:44:54.420611 (+ 428us) server_negotiation.cc:299] Negotiation successful
0925 22:44:54.420765 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260925 22:44:54.423062 18116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:44:54.384202 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45654 (local address 127.10.198.60:44389)
0925 22:44:54.384347 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0925 22:44:54.384352 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:44:54.412168 (+ 27816us) server_negotiation.cc:407] Connection header received
0925 22:44:54.412407 (+ 239us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:44:54.412411 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:44:54.412479 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:44:54.412545 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:44:54.413541 (+ 996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:44:54.414428 (+ 887us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:44:54.415695 (+ 1267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:44:54.416071 (+ 376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:44:54.416151 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:44:54.418377 (+ 2226us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:44:54.418382 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:44:54.418385 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:44:54.418413 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:44:54.420650 (+ 2237us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:44:54.421395 (+ 745us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:44:54.421399 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:44:54.421401 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:44:54.421513 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:44:54.421940 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:44:54.421944 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:44:54.421946 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:44:54.422210 (+ 264us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:44:54.422352 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0925 22:44:54.422688 (+ 336us) server_negotiation.cc:299] Negotiation successful
0925 22:44:54.422818 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260925 22:44:55.360235 18451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:44:55.359681 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.198.62:41969 (local address 127.10.198.2:60435)
0925 22:44:55.359974 (+ 293us) negotiation.cc:107] Waiting for socket to connect
0925 22:44:55.360097 (+ 123us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.198.62:41969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":201,"thread_start_us":88,"threads_started":1}
W20260925 22:44:55.360598 18248 heartbeater.cc:644] Failed to heartbeat to 127.10.198.62:41969 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.10.198.62:41969: connect: Connection refused (error 111)
I20260925 22:44:55.383121 18452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:44:55.382649 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.198.62:41969 (local address 127.10.198.3:57169)
0925 22:44:55.382927 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0925 22:44:55.383021 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.198.62:41969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":199,"thread_start_us":76,"threads_started":1}
W20260925 22:44:55.383682 18385 heartbeater.cc:644] Failed to heartbeat to 127.10.198.62:41969 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.10.198.62:41969: connect: Connection refused (error 111)
I20260925 22:44:55.386178 18453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:44:55.385632 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.198.62:41969 (local address 127.10.198.1:40257)
0925 22:44:55.385950 (+ 318us) negotiation.cc:107] Waiting for socket to connect
0925 22:44:55.386037 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.198.62:41969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":216,"thread_start_us":73,"threads_started":1}
W20260925 22:44:55.386544 18111 heartbeater.cc:644] Failed to heartbeat to 127.10.198.62:41969 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.10.198.62:41969: connect: Connection refused (error 111)
I20260925 22:44:55.875128 18454 raft_consensus.cc:482] T 00000000000000000000000000000000 P f7d2a2767543443e8ba59b91bd97f4e1 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 36158ad8c29246c2ab5424b4baf3038d)
I20260925 22:44:55.875351 18454 raft_consensus.cc:504] T 00000000000000000000000000000000 P f7d2a2767543443e8ba59b91bd97f4e1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "36158ad8c29246c2ab5424b4baf3038d" member_type: VOTER last_known_addr { host: "127.10.198.62" port: 41969 } } peers { permanent_uuid: "f7d2a2767543443e8ba59b91bd97f4e1" member_type: VOTER last_known_addr { host: "127.10.198.61" port: 39135 } attrs { promote: false } } peers { permanent_uuid: "ed073dd0b5e2464bbc701bc2862011ea" member_type: VOTER last_known_addr { host: "127.10.198.60" port: 44389 } attrs { promote: false } }
I20260925 22:44:55.876655 18454 leader_election.cc:288] T 00000000000000000000000000000000 P f7d2a2767543443e8ba59b91bd97f4e1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 36158ad8c29246c2ab5424b4baf3038d (127.10.198.62:41969), ed073dd0b5e2464bbc701bc2862011ea (127.10.198.60:44389)
I20260925 22:44:55.877558 18455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:44:55.876944 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.198.62:41969 (local address 127.0.0.1:42526)
0925 22:44:55.877314 (+ 370us) negotiation.cc:107] Waiting for socket to connect
0925 22:44:55.877408 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.198.62:41969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":283,"mutex_wait_us":53,"thread_start_us":61,"threads_started":1}
W20260925 22:44:55.877835 17734 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.10.198.62:41969: connect: Connection refused (error 111)
I20260925 22:44:55.878762 18459 raft_consensus.cc:482] T 00000000000000000000000000000000 P ed073dd0b5e2464bbc701bc2862011ea [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 36158ad8c29246c2ab5424b4baf3038d)
I20260925 22:44:55.878835 18455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:44:55.878548 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.198.62:41969 (local address 127.0.0.1:42528)
0925 22:44:55.878697 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0925 22:44:55.878758 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.198.62:41969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
W20260925 22:44:55.879014 17734 leader_election.cc:334] T 00000000000000000000000000000000 P f7d2a2767543443e8ba59b91bd97f4e1 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 36158ad8c29246c2ab5424b4baf3038d (127.10.198.62:41969): Network error: Client connection negotiation failed: client connection to 127.10.198.62:41969: connect: Connection refused (error 111)
I20260925 22:44:55.879431 18459 raft_consensus.cc:504] T 00000000000000000000000000000000 P ed073dd0b5e2464bbc701bc2862011ea [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "36158ad8c29246c2ab5424b4baf3038d" member_type: VOTER last_known_addr { host: "127.10.198.62" port: 41969 } } peers { permanent_uuid: "f7d2a2767543443e8ba59b91bd97f4e1" member_type: VOTER last_known_addr { host: "127.10.198.61" port: 39135 } attrs { promote: false } } peers { permanent_uuid: "ed073dd0b5e2464bbc701bc2862011ea" member_type: VOTER last_known_addr { host: "127.10.198.60" port: 44389 } attrs { promote: false } }
I20260925 22:44:55.881137 18459 leader_election.cc:288] T 00000000000000000000000000000000 P ed073dd0b5e2464bbc701bc2862011ea [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 36158ad8c29246c2ab5424b4baf3038d (127.10.198.62:41969), f7d2a2767543443e8ba59b91bd97f4e1 (127.10.198.61:39135)
I20260925 22:44:55.881883 18460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:44:55.881288 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.198.62:41969 (local address 127.0.0.1:42536)
0925 22:44:55.881662 (+ 374us) negotiation.cc:107] Waiting for socket to connect
0925 22:44:55.881749 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.198.62:41969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":299,"thread_start_us":78,"threads_started":1}
W20260925 22:44:55.882226 17917 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.10.198.62:41969: connect: Connection refused (error 111)
I20260925 22:44:55.882402 18456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:44:55.876944 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.198.60:44389 (local address 127.0.0.1:45674)
0925 22:44:55.877277 (+ 333us) negotiation.cc:107] Waiting for socket to connect
0925 22:44:55.877296 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0925 22:44:55.877466 (+ 170us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:44:55.877792 (+ 326us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:44:55.877795 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:44:55.877812 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:44:55.877917 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:44:55.877931 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:44:55.879030 (+ 1099us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:44:55.879036 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:44:55.880686 (+ 1650us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:44:55.880703 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:44:55.881913 (+ 1210us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:44:55.881918 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:44:55.882046 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:44:55.882085 (+ 39us) client_negotiation.cc:769] Sending connection context
0925 22:44:55.882146 (+ 61us) client_negotiation.cc:240] Negotiation successful
0925 22:44:55.882248 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":238,"thread_start_us":90,"threads_started":1}
I20260925 22:44:55.883018 18457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:44:55.877174 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45674 (local address 127.10.198.60:44389)
0925 22:44:55.877416 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0925 22:44:55.877420 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:44:55.877439 (+ 19us) server_negotiation.cc:407] Connection header received
0925 22:44:55.877584 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:44:55.877588 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:44:55.877662 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:44:55.877771 (+ 109us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:44:55.878078 (+ 307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:44:55.878884 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:44:55.880909 (+ 2025us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:44:55.881746 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:44:55.882758 (+ 1012us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:44:55.882812 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0925 22:44:55.882851 (+ 39us) server_negotiation.cc:299] Negotiation successful
0925 22:44:55.882927 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":167,"thread_start_us":69,"threads_started":1}
I20260925 22:44:55.883512 17941 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f7d2a2767543443e8ba59b91bd97f4e1" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "ed073dd0b5e2464bbc701bc2862011ea" is_pre_election: true
I20260925 22:44:55.883792 17941 raft_consensus.cc:2454] T 00000000000000000000000000000000 P ed073dd0b5e2464bbc701bc2862011ea [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f7d2a2767543443e8ba59b91bd97f4e1 in term 1.
I20260925 22:44:55.883999 18460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:44:55.883703 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.198.62:41969 (local address 127.0.0.1:42550)
0925 22:44:55.883882 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0925 22:44:55.883934 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.198.62:41969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":92}
W20260925 22:44:55.884099 17917 leader_election.cc:334] T 00000000000000000000000000000000 P ed073dd0b5e2464bbc701bc2862011ea [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 36158ad8c29246c2ab5424b4baf3038d (127.10.198.62:41969): Network error: Client connection negotiation failed: client connection to 127.10.198.62:41969: connect: Connection refused (error 111)
I20260925 22:44:55.884243 17733 leader_election.cc:302] T 00000000000000000000000000000000 P f7d2a2767543443e8ba59b91bd97f4e1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ed073dd0b5e2464bbc701bc2862011ea, f7d2a2767543443e8ba59b91bd97f4e1; no voters: 36158ad8c29246c2ab5424b4baf3038d
I20260925 22:44:55.884577 18454 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f7d2a2767543443e8ba59b91bd97f4e1 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260925 22:44:55.884650 18454 raft_consensus.cc:482] T 00000000000000000000000000000000 P f7d2a2767543443e8ba59b91bd97f4e1 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 36158ad8c29246c2ab5424b4baf3038d)
I20260925 22:44:55.884694 18454 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f7d2a2767543443e8ba59b91bd97f4e1 [term 1 FOLLOWER]: Advancing to term 2
I20260925 22:44:55.886261 18454 raft_consensus.cc:504] T 00000000000000000000000000000000 P f7d2a2767543443e8ba59b91bd97f4e1 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "36158ad8c29246c2ab5424b4baf3038d" member_type: VOTER last_known_addr { host: "127.10.198.62" port: 41969 } } peers { permanent_uuid: "f7d2a2767543443e8ba59b91bd97f4e1" member_type: VOTER last_known_addr { host: "127.10.198.61" port: 39135 } attrs { promote: false } } peers { permanent_uuid: "ed073dd0b5e2464bbc701bc2862011ea" member_type: VOTER last_known_addr { host: "127.10.198.60" port: 44389 } attrs { promote: false } }
I20260925 22:44:55.886803 18454 leader_election.cc:288] T 00000000000000000000000000000000 P f7d2a2767543443e8ba59b91bd97f4e1 [CANDIDATE]: Term 2 election: Requested vote from peers 36158ad8c29246c2ab5424b4baf3038d (127.10.198.62:41969), ed073dd0b5e2464bbc701bc2862011ea (127.10.198.60:44389)
I20260925 22:44:55.887017 18461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:44:55.881354 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.198.61:39135 (local address 127.0.0.1:59336)
0925 22:44:55.881844 (+ 490us) negotiation.cc:107] Waiting for socket to connect
0925 22:44:55.881861 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 22:44:55.882226 (+ 365us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:44:55.882807 (+ 581us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:44:55.882811 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:44:55.882831 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:44:55.883069 (+ 238us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:44:55.883083 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:44:55.884307 (+ 1224us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:44:55.884311 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:44:55.885503 (+ 1192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:44:55.885574 (+ 71us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:44:55.886424 (+ 850us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:44:55.886428 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:44:55.886518 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:44:55.886545 (+ 27us) client_negotiation.cc:769] Sending connection context
0925 22:44:55.886736 (+ 191us) client_negotiation.cc:240] Negotiation successful
0925 22:44:55.886836 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":421,"thread_start_us":67,"threads_started":1}
I20260925 22:44:55.887027 18462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:44:55.881514 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59336 (local address 127.10.198.61:39135)
0925 22:44:55.881847 (+ 333us) server_negotiation.cc:206] Beginning negotiation
0925 22:44:55.881859 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 22:44:55.882258 (+ 399us) server_negotiation.cc:407] Connection header received
0925 22:44:55.882419 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:44:55.882422 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:44:55.882497 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:44:55.882615 (+ 118us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:44:55.883310 (+ 695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:44:55.884159 (+ 849us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:44:55.885701 (+ 1542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:44:55.886287 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:44:55.886378 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:44:55.886472 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0925 22:44:55.886787 (+ 315us) server_negotiation.cc:299] Negotiation successful
0925 22:44:55.886896 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":255,"thread_start_us":139,"threads_started":1}
I20260925 22:44:55.887315 17941 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f7d2a2767543443e8ba59b91bd97f4e1" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "ed073dd0b5e2464bbc701bc2862011ea"
I20260925 22:44:55.887449 17941 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ed073dd0b5e2464bbc701bc2862011ea [term 1 FOLLOWER]: Advancing to term 2
I20260925 22:44:55.887555 18456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:44:55.886965 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.198.62:41969 (local address 127.0.0.1:42564)
0925 22:44:55.887416 (+ 451us) negotiation.cc:107] Waiting for socket to connect
0925 22:44:55.887481 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.198.62:41969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":375}
I20260925 22:44:55.887607 17758 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ed073dd0b5e2464bbc701bc2862011ea" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "f7d2a2767543443e8ba59b91bd97f4e1" is_pre_election: true
I20260925 22:44:55.887889 17758 raft_consensus.cc:2379] T 00000000000000000000000000000000 P f7d2a2767543443e8ba59b91bd97f4e1 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ed073dd0b5e2464bbc701bc2862011ea in current term 2: Already voted for candidate f7d2a2767543443e8ba59b91bd97f4e1 in this term.
I20260925 22:44:55.888404 18456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:44:55.888104 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.198.62:41969 (local address 127.0.0.1:42576)
0925 22:44:55.888268 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0925 22:44:55.888329 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.198.62:41969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97}
I20260925 22:44:55.888384 17915 leader_election.cc:302] T 00000000000000000000000000000000 P ed073dd0b5e2464bbc701bc2862011ea [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ed073dd0b5e2464bbc701bc2862011ea; no voters: 36158ad8c29246c2ab5424b4baf3038d, f7d2a2767543443e8ba59b91bd97f4e1
W20260925 22:44:55.888536 17734 leader_election.cc:334] T 00000000000000000000000000000000 P f7d2a2767543443e8ba59b91bd97f4e1 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 36158ad8c29246c2ab5424b4baf3038d (127.10.198.62:41969): Network error: Client connection negotiation failed: client connection to 127.10.198.62:41969: connect: Connection refused (error 111)
I20260925 22:44:55.888938 17941 raft_consensus.cc:2454] T 00000000000000000000000000000000 P ed073dd0b5e2464bbc701bc2862011ea [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f7d2a2767543443e8ba59b91bd97f4e1 in term 2.
I20260925 22:44:55.889140 18459 raft_consensus.cc:2726] T 00000000000000000000000000000000 P ed073dd0b5e2464bbc701bc2862011ea [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 22:44:55.889417 17733 leader_election.cc:302] T 00000000000000000000000000000000 P f7d2a2767543443e8ba59b91bd97f4e1 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ed073dd0b5e2464bbc701bc2862011ea, f7d2a2767543443e8ba59b91bd97f4e1; no voters: 36158ad8c29246c2ab5424b4baf3038d
I20260925 22:44:55.889693 18454 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f7d2a2767543443e8ba59b91bd97f4e1 [term 2 FOLLOWER]: Leader election won for term 2
I20260925 22:44:55.889918 18454 raft_consensus.cc:686] T 00000000000000000000000000000000 P f7d2a2767543443e8ba59b91bd97f4e1 [term 2 LEADER]: Becoming Leader. State: Replica: f7d2a2767543443e8ba59b91bd97f4e1, State: Running, Role: LEADER
I20260925 22:44:55.890136 18454 consensus_queue.cc:237] T 00000000000000000000000000000000 P f7d2a2767543443e8ba59b91bd97f4e1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "36158ad8c29246c2ab5424b4baf3038d" member_type: VOTER last_known_addr { host: "127.10.198.62" port: 41969 } } peers { permanent_uuid: "f7d2a2767543443e8ba59b91bd97f4e1" member_type: VOTER last_known_addr { host: "127.10.198.61" port: 39135 } attrs { promote: false } } peers { permanent_uuid: "ed073dd0b5e2464bbc701bc2862011ea" member_type: VOTER last_known_addr { host: "127.10.198.60" port: 44389 } attrs { promote: false } }
I20260925 22:44:55.891527 18464 sys_catalog.cc:455] T 00000000000000000000000000000000 P f7d2a2767543443e8ba59b91bd97f4e1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f7d2a2767543443e8ba59b91bd97f4e1. Latest consensus state: current_term: 2 leader_uuid: "f7d2a2767543443e8ba59b91bd97f4e1" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "36158ad8c29246c2ab5424b4baf3038d" member_type: VOTER last_known_addr { host: "127.10.198.62" port: 41969 } } peers { permanent_uuid: "f7d2a2767543443e8ba59b91bd97f4e1" member_type: VOTER last_known_addr { host: "127.10.198.61" port: 39135 } attrs { promote: false } } peers { permanent_uuid: "ed073dd0b5e2464bbc701bc2862011ea" member_type: VOTER last_known_addr { host: "127.10.198.60" port: 44389 } attrs { promote: false } } }
I20260925 22:44:55.891680 18464 sys_catalog.cc:458] T 00000000000000000000000000000000 P f7d2a2767543443e8ba59b91bd97f4e1 [sys.catalog]: This master's current role is: LEADER
I20260925 22:44:55.892042 18466 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 22:44:55.894332 18466 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 22:44:55.894739 18466 catalog_manager.cc:1259] Loaded cluster ID: dec13579c22b435db994a21e4ce3e494
I20260925 22:44:55.894786 18466 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 22:44:55.895306 18466 catalog_manager.cc:1504] Loading token signing keys...
I20260925 22:44:55.895648 18466 catalog_manager.cc:5879] T 00000000000000000000000000000000 P f7d2a2767543443e8ba59b91bd97f4e1: Loaded TSK: 0
I20260925 22:44:55.896489 18466 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 22:44:55.947662 17748 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59322:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 22:44:55.951517 17748 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 22:44:55.954988 18456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:44:55.954682 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.198.62:41969 (local address 127.0.0.1:42608)
0925 22:44:55.954813 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0925 22:44:55.954890 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.198.62:41969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260925 22:44:55.955119 17941 raft_consensus.cc:1261] T 00000000000000000000000000000000 P ed073dd0b5e2464bbc701bc2862011ea [term 2 FOLLOWER]: Refusing update from remote peer f7d2a2767543443e8ba59b91bd97f4e1: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 2 index: 14. (index mismatch)
I20260925 22:44:55.955775 18464 consensus_queue.cc:1035] T 00000000000000000000000000000000 P f7d2a2767543443e8ba59b91bd97f4e1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ed073dd0b5e2464bbc701bc2862011ea" member_type: VOTER last_known_addr { host: "127.10.198.60" port: 44389 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260925 22:44:55.956061 18456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:44:55.955790 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.198.62:41969 (local address 127.0.0.1:42610)
0925 22:44:55.955937 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0925 22:44:55.955996 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.198.62:41969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260925 22:44:55.956190 17734 consensus_peers.cc:487] T 00000000000000000000000000000000 P f7d2a2767543443e8ba59b91bd97f4e1 -> Peer 36158ad8c29246c2ab5424b4baf3038d (127.10.198.62:41969): Couldn't send request to peer 36158ad8c29246c2ab5424b4baf3038d. Status: Network error: Client connection negotiation failed: client connection to 127.10.198.62:41969: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260925 22:44:55.957492 18459 sys_catalog.cc:455] T 00000000000000000000000000000000 P ed073dd0b5e2464bbc701bc2862011ea [sys.catalog]: SysCatalogTable state changed. Reason: New leader f7d2a2767543443e8ba59b91bd97f4e1. Latest consensus state: current_term: 2 leader_uuid: "f7d2a2767543443e8ba59b91bd97f4e1" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "36158ad8c29246c2ab5424b4baf3038d" member_type: VOTER last_known_addr { host: "127.10.198.62" port: 41969 } } peers { permanent_uuid: "f7d2a2767543443e8ba59b91bd97f4e1" member_type: VOTER last_known_addr { host: "127.10.198.61" port: 39135 } attrs { promote: false } } peers { permanent_uuid: "ed073dd0b5e2464bbc701bc2862011ea" member_type: VOTER last_known_addr { host: "127.10.198.60" port: 44389 } attrs { promote: false } } }
I20260925 22:44:55.957659 18459 sys_catalog.cc:458] T 00000000000000000000000000000000 P ed073dd0b5e2464bbc701bc2862011ea [sys.catalog]: This master's current role is: FOLLOWER
I20260925 22:44:55.958878 18464 sys_catalog.cc:455] T 00000000000000000000000000000000 P f7d2a2767543443e8ba59b91bd97f4e1 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "f7d2a2767543443e8ba59b91bd97f4e1" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "36158ad8c29246c2ab5424b4baf3038d" member_type: VOTER last_known_addr { host: "127.10.198.62" port: 41969 } } peers { permanent_uuid: "f7d2a2767543443e8ba59b91bd97f4e1" member_type: VOTER last_known_addr { host: "127.10.198.61" port: 39135 } attrs { promote: false } } peers { permanent_uuid: "ed073dd0b5e2464bbc701bc2862011ea" member_type: VOTER last_known_addr { host: "127.10.198.60" port: 44389 } attrs { promote: false } } }
I20260925 22:44:55.959012 18464 sys_catalog.cc:458] T 00000000000000000000000000000000 P f7d2a2767543443e8ba59b91bd97f4e1 [sys.catalog]: This master's current role is: LEADER
I20260925 22:44:55.959404 18459 sys_catalog.cc:455] T 00000000000000000000000000000000 P ed073dd0b5e2464bbc701bc2862011ea [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "f7d2a2767543443e8ba59b91bd97f4e1" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "36158ad8c29246c2ab5424b4baf3038d" member_type: VOTER last_known_addr { host: "127.10.198.62" port: 41969 } } peers { permanent_uuid: "f7d2a2767543443e8ba59b91bd97f4e1" member_type: VOTER last_known_addr { host: "127.10.198.61" port: 39135 } attrs { promote: false } } peers { permanent_uuid: "ed073dd0b5e2464bbc701bc2862011ea" member_type: VOTER last_known_addr { host: "127.10.198.60" port: 44389 } attrs { promote: false } } }
I20260925 22:44:55.959535 18459 sys_catalog.cc:458] T 00000000000000000000000000000000 P ed073dd0b5e2464bbc701bc2862011ea [sys.catalog]: This master's current role is: FOLLOWER
I20260925 22:44:55.970752 18473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:44:55.965720 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44700 (local address 127.10.198.3:46715)
0925 22:44:55.966468 (+ 748us) server_negotiation.cc:206] Beginning negotiation
0925 22:44:55.966476 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 22:44:55.966493 (+ 17us) server_negotiation.cc:407] Connection header received
0925 22:44:55.966546 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:44:55.966549 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:44:55.966602 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:44:55.966968 (+ 366us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:44:55.967077 (+ 109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:44:55.967968 (+ 891us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:44:55.969374 (+ 1406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:44:55.970026 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:44:55.970319 (+ 293us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:44:55.970385 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0925 22:44:55.970493 (+ 108us) server_negotiation.cc:299] Negotiation successful
0925 22:44:55.970579 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":662,"thread_start_us":85,"threads_started":1}
I20260925 22:44:55.970758 18456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:44:55.965610 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.198.3:46715 (local address 127.0.0.1:44700)
0925 22:44:55.966027 (+ 417us) negotiation.cc:107] Waiting for socket to connect
0925 22:44:55.966043 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 22:44:55.966124 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:44:55.966727 (+ 603us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:44:55.966731 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:44:55.966747 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:44:55.966860 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:44:55.966865 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:44:55.968078 (+ 1213us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:44:55.968082 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:44:55.969237 (+ 1155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:44:55.969242 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:44:55.970166 (+ 924us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:44:55.970170 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:44:55.970313 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:44:55.970401 (+ 88us) client_negotiation.cc:769] Sending connection context
0925 22:44:55.970475 (+ 74us) client_negotiation.cc:240] Negotiation successful
0925 22:44:55.970612 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":341,"mutex_wait_us":109}
I20260925 22:44:55.970897 18476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:44:55.966426 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42512 (local address 127.10.198.2:35073)
0925 22:44:55.966660 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0925 22:44:55.966666 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 22:44:55.966696 (+ 30us) server_negotiation.cc:407] Connection header received
0925 22:44:55.966793 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:44:55.966795 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:44:55.966844 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:44:55.966938 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:44:55.967176 (+ 238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:44:55.967712 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:44:55.969374 (+ 1662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:44:55.970026 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:44:55.970142 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:44:55.970225 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0925 22:44:55.970674 (+ 449us) server_negotiation.cc:299] Negotiation successful
0925 22:44:55.970750 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":161,"thread_start_us":72,"threads_started":1}
I20260925 22:44:55.971243 18455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:44:55.966029 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.198.1:41135 (local address 127.0.0.1:48094)
0925 22:44:55.966334 (+ 305us) negotiation.cc:107] Waiting for socket to connect
0925 22:44:55.966348 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 22:44:55.966435 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:44:55.967015 (+ 580us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:44:55.967017 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:44:55.967033 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:44:55.967111 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:44:55.967118 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:44:55.968217 (+ 1099us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:44:55.968220 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:44:55.969236 (+ 1016us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:44:55.969244 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:44:55.970149 (+ 905us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:44:55.970153 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:44:55.970253 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:44:55.970390 (+ 137us) client_negotiation.cc:769] Sending connection context
0925 22:44:55.970468 (+ 78us) client_negotiation.cc:240] Negotiation successful
0925 22:44:55.971112 (+ 644us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":235,"mutex_wait_us":617}
I20260925 22:44:55.970799 18475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:44:55.966130 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48094 (local address 127.10.198.1:41135)
0925 22:44:55.966782 (+ 652us) server_negotiation.cc:206] Beginning negotiation
0925 22:44:55.966786 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:44:55.966799 (+ 13us) server_negotiation.cc:407] Connection header received
0925 22:44:55.966846 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:44:55.966848 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:44:55.966893 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:44:55.967209 (+ 316us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:44:55.967314 (+ 105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:44:55.968092 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:44:55.969376 (+ 1284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:44:55.970015 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:44:55.970145 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:44:55.970223 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0925 22:44:55.970573 (+ 350us) server_negotiation.cc:299] Negotiation successful
0925 22:44:55.970660 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":574,"thread_start_us":74,"threads_started":1}
I20260925 22:44:55.971540 18474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:44:55.966069 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.198.2:35073 (local address 127.0.0.1:42512)
0925 22:44:55.966616 (+ 547us) negotiation.cc:107] Waiting for socket to connect
0925 22:44:55.966629 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 22:44:55.966705 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:44:55.966943 (+ 238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:44:55.966946 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:44:55.966961 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:44:55.967062 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:44:55.967067 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:44:55.967826 (+ 759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:44:55.967832 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:44:55.969214 (+ 1382us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:44:55.969227 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:44:55.970277 (+ 1050us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:44:55.970282 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:44:55.970422 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:44:55.970445 (+ 23us) client_negotiation.cc:769] Sending connection context
0925 22:44:55.970519 (+ 74us) client_negotiation.cc:240] Negotiation successful
0925 22:44:55.970602 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":492,"mutex_wait_us":64,"thread_start_us":112,"threads_started":1}
I20260925 22:44:55.971545 18318 tablet_service.cc:1467] Processing CreateTablet for tablet 8ade2068da5b404ea572164cf498c942 (DEFAULT_TABLE table=test-table [id=3aff0a87d5984c43848c1400a31a30a5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 22:44:55.971958 18318 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8ade2068da5b404ea572164cf498c942. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:44:55.972098 18181 tablet_service.cc:1467] Processing CreateTablet for tablet 8ade2068da5b404ea572164cf498c942 (DEFAULT_TABLE table=test-table [id=3aff0a87d5984c43848c1400a31a30a5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 22:44:55.972110 18044 tablet_service.cc:1467] Processing CreateTablet for tablet 8ade2068da5b404ea572164cf498c942 (DEFAULT_TABLE table=test-table [id=3aff0a87d5984c43848c1400a31a30a5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 22:44:55.972344 18181 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8ade2068da5b404ea572164cf498c942. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:44:55.972376 18044 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8ade2068da5b404ea572164cf498c942. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:44:55.974937 18477 tablet_bootstrap.cc:492] T 8ade2068da5b404ea572164cf498c942 P f5cc83f491e5409895f3eaf63bdc05c6: Bootstrap starting.
I20260925 22:44:55.975260 18478 tablet_bootstrap.cc:492] T 8ade2068da5b404ea572164cf498c942 P 1ffb9543e6c340f78a7f82ffff34faf9: Bootstrap starting.
I20260925 22:44:55.975363 18479 tablet_bootstrap.cc:492] T 8ade2068da5b404ea572164cf498c942 P dcfd1608a17742d9a03fa549b12bb0c8: Bootstrap starting.
I20260925 22:44:55.976002 18477 tablet_bootstrap.cc:654] T 8ade2068da5b404ea572164cf498c942 P f5cc83f491e5409895f3eaf63bdc05c6: Neither blocks nor log segments found. Creating new log.
I20260925 22:44:55.976228 18478 tablet_bootstrap.cc:654] T 8ade2068da5b404ea572164cf498c942 P 1ffb9543e6c340f78a7f82ffff34faf9: Neither blocks nor log segments found. Creating new log.
I20260925 22:44:55.976383 18479 tablet_bootstrap.cc:654] T 8ade2068da5b404ea572164cf498c942 P dcfd1608a17742d9a03fa549b12bb0c8: Neither blocks nor log segments found. Creating new log.
I20260925 22:44:55.977475 18479 tablet_bootstrap.cc:492] T 8ade2068da5b404ea572164cf498c942 P dcfd1608a17742d9a03fa549b12bb0c8: No bootstrap required, opened a new log
I20260925 22:44:55.977557 18479 ts_tablet_manager.cc:1397] T 8ade2068da5b404ea572164cf498c942 P dcfd1608a17742d9a03fa549b12bb0c8: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260925 22:44:55.977589 18477 tablet_bootstrap.cc:492] T 8ade2068da5b404ea572164cf498c942 P f5cc83f491e5409895f3eaf63bdc05c6: No bootstrap required, opened a new log
I20260925 22:44:55.977679 18477 ts_tablet_manager.cc:1397] T 8ade2068da5b404ea572164cf498c942 P f5cc83f491e5409895f3eaf63bdc05c6: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20260925 22:44:55.978040 18479 raft_consensus.cc:348] T 8ade2068da5b404ea572164cf498c942 P dcfd1608a17742d9a03fa549b12bb0c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ffb9543e6c340f78a7f82ffff34faf9" member_type: VOTER last_known_addr { host: "127.10.198.3" port: 46715 } } peers { permanent_uuid: "dcfd1608a17742d9a03fa549b12bb0c8" member_type: VOTER last_known_addr { host: "127.10.198.1" port: 41135 } } peers { permanent_uuid: "f5cc83f491e5409895f3eaf63bdc05c6" member_type: VOTER last_known_addr { host: "127.10.198.2" port: 35073 } }
I20260925 22:44:55.978047 18477 raft_consensus.cc:348] T 8ade2068da5b404ea572164cf498c942 P f5cc83f491e5409895f3eaf63bdc05c6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ffb9543e6c340f78a7f82ffff34faf9" member_type: VOTER last_known_addr { host: "127.10.198.3" port: 46715 } } peers { permanent_uuid: "dcfd1608a17742d9a03fa549b12bb0c8" member_type: VOTER last_known_addr { host: "127.10.198.1" port: 41135 } } peers { permanent_uuid: "f5cc83f491e5409895f3eaf63bdc05c6" member_type: VOTER last_known_addr { host: "127.10.198.2" port: 35073 } }
I20260925 22:44:55.978147 18479 raft_consensus.cc:374] T 8ade2068da5b404ea572164cf498c942 P dcfd1608a17742d9a03fa549b12bb0c8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:44:55.978171 18477 raft_consensus.cc:374] T 8ade2068da5b404ea572164cf498c942 P f5cc83f491e5409895f3eaf63bdc05c6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:44:55.978169 18479 raft_consensus.cc:729] T 8ade2068da5b404ea572164cf498c942 P dcfd1608a17742d9a03fa549b12bb0c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dcfd1608a17742d9a03fa549b12bb0c8, State: Initialized, Role: FOLLOWER
I20260925 22:44:55.978229 18477 raft_consensus.cc:729] T 8ade2068da5b404ea572164cf498c942 P f5cc83f491e5409895f3eaf63bdc05c6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f5cc83f491e5409895f3eaf63bdc05c6, State: Initialized, Role: FOLLOWER
I20260925 22:44:55.978361 18479 consensus_queue.cc:260] T 8ade2068da5b404ea572164cf498c942 P dcfd1608a17742d9a03fa549b12bb0c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ffb9543e6c340f78a7f82ffff34faf9" member_type: VOTER last_known_addr { host: "127.10.198.3" port: 46715 } } peers { permanent_uuid: "dcfd1608a17742d9a03fa549b12bb0c8" member_type: VOTER last_known_addr { host: "127.10.198.1" port: 41135 } } peers { permanent_uuid: "f5cc83f491e5409895f3eaf63bdc05c6" member_type: VOTER last_known_addr { host: "127.10.198.2" port: 35073 } }
I20260925 22:44:55.978370 18477 consensus_queue.cc:260] T 8ade2068da5b404ea572164cf498c942 P f5cc83f491e5409895f3eaf63bdc05c6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ffb9543e6c340f78a7f82ffff34faf9" member_type: VOTER last_known_addr { host: "127.10.198.3" port: 46715 } } peers { permanent_uuid: "dcfd1608a17742d9a03fa549b12bb0c8" member_type: VOTER last_known_addr { host: "127.10.198.1" port: 41135 } } peers { permanent_uuid: "f5cc83f491e5409895f3eaf63bdc05c6" member_type: VOTER last_known_addr { host: "127.10.198.2" port: 35073 } }
I20260925 22:44:55.978724 18479 ts_tablet_manager.cc:1428] T 8ade2068da5b404ea572164cf498c942 P dcfd1608a17742d9a03fa549b12bb0c8: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260925 22:44:55.978761 18477 ts_tablet_manager.cc:1428] T 8ade2068da5b404ea572164cf498c942 P f5cc83f491e5409895f3eaf63bdc05c6: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260925 22:44:55.980465 18486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:44:55.979909 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.198.62:41969 (local address 127.10.198.1:38453)
0925 22:44:55.980253 (+ 344us) negotiation.cc:107] Waiting for socket to connect
0925 22:44:55.980322 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.198.62:41969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":270,"thread_start_us":103,"threads_started":1}
I20260925 22:44:55.981850 18478 tablet_bootstrap.cc:492] T 8ade2068da5b404ea572164cf498c942 P 1ffb9543e6c340f78a7f82ffff34faf9: No bootstrap required, opened a new log
I20260925 22:44:55.981966 18478 ts_tablet_manager.cc:1397] T 8ade2068da5b404ea572164cf498c942 P 1ffb9543e6c340f78a7f82ffff34faf9: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20260925 22:44:55.982010 18249 heartbeater.cc:503] Master 127.10.198.61:39135 was elected leader, sending a full tablet report...
I20260925 22:44:55.982096 18112 heartbeater.cc:503] Master 127.10.198.61:39135 was elected leader, sending a full tablet report...
I20260925 22:44:55.982513 18478 raft_consensus.cc:348] T 8ade2068da5b404ea572164cf498c942 P 1ffb9543e6c340f78a7f82ffff34faf9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ffb9543e6c340f78a7f82ffff34faf9" member_type: VOTER last_known_addr { host: "127.10.198.3" port: 46715 } } peers { permanent_uuid: "dcfd1608a17742d9a03fa549b12bb0c8" member_type: VOTER last_known_addr { host: "127.10.198.1" port: 41135 } } peers { permanent_uuid: "f5cc83f491e5409895f3eaf63bdc05c6" member_type: VOTER last_known_addr { host: "127.10.198.2" port: 35073 } }
I20260925 22:44:55.982640 18478 raft_consensus.cc:374] T 8ade2068da5b404ea572164cf498c942 P 1ffb9543e6c340f78a7f82ffff34faf9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:44:55.982672 18478 raft_consensus.cc:729] T 8ade2068da5b404ea572164cf498c942 P 1ffb9543e6c340f78a7f82ffff34faf9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1ffb9543e6c340f78a7f82ffff34faf9, State: Initialized, Role: FOLLOWER
I20260925 22:44:55.982880 18485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:44:55.979082 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.198.62:41969 (local address 127.10.198.2:46481)
0925 22:44:55.982620 (+ 3538us) negotiation.cc:107] Waiting for socket to connect
0925 22:44:55.982706 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.198.62:41969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":3459,"thread_start_us":98,"threads_started":1}
I20260925 22:44:55.982817 18478 consensus_queue.cc:260] T 8ade2068da5b404ea572164cf498c942 P 1ffb9543e6c340f78a7f82ffff34faf9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ffb9543e6c340f78a7f82ffff34faf9" member_type: VOTER last_known_addr { host: "127.10.198.3" port: 46715 } } peers { permanent_uuid: "dcfd1608a17742d9a03fa549b12bb0c8" member_type: VOTER last_known_addr { host: "127.10.198.1" port: 41135 } } peers { permanent_uuid: "f5cc83f491e5409895f3eaf63bdc05c6" member_type: VOTER last_known_addr { host: "127.10.198.2" port: 35073 } }
I20260925 22:44:55.983723 18478 ts_tablet_manager.cc:1428] T 8ade2068da5b404ea572164cf498c942 P 1ffb9543e6c340f78a7f82ffff34faf9: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260925 22:44:55.983865 18485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:44:55.983347 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.198.62:41969 (local address 127.10.198.2:48377)
0925 22:44:55.983701 (+ 354us) negotiation.cc:107] Waiting for socket to connect
0925 22:44:55.983757 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.198.62:41969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":298,"thread_start_us":151,"threads_started":1}
I20260925 22:44:55.984059 18486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:44:55.982330 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.198.62:41969 (local address 127.10.198.1:43337)
0925 22:44:55.982956 (+ 626us) negotiation.cc:107] Waiting for socket to connect
0925 22:44:55.983011 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.198.62:41969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
I20260925 22:44:55.985006 18489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:44:55.984536 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.198.62:41969 (local address 127.10.198.3:53209)
0925 22:44:55.984774 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0925 22:44:55.984845 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.198.62:41969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":166,"thread_start_us":102,"threads_started":1}
I20260925 22:44:55.985893 18386 heartbeater.cc:503] Master 127.10.198.61:39135 was elected leader, sending a full tablet report...
I20260925 22:44:55.986429 18489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:44:55.985489 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.198.62:41969 (local address 127.10.198.3:34877)
0925 22:44:55.985784 (+ 295us) negotiation.cc:107] Waiting for socket to connect
0925 22:44:55.986348 (+ 564us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.198.62:41969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":237,"thread_start_us":99,"threads_started":1}
I20260925 22:44:56.000140 18487 raft_consensus.cc:482] T 8ade2068da5b404ea572164cf498c942 P 1ffb9543e6c340f78a7f82ffff34faf9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 22:44:56.000306 18487 raft_consensus.cc:504] T 8ade2068da5b404ea572164cf498c942 P 1ffb9543e6c340f78a7f82ffff34faf9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ffb9543e6c340f78a7f82ffff34faf9" member_type: VOTER last_known_addr { host: "127.10.198.3" port: 46715 } } peers { permanent_uuid: "dcfd1608a17742d9a03fa549b12bb0c8" member_type: VOTER last_known_addr { host: "127.10.198.1" port: 41135 } } peers { permanent_uuid: "f5cc83f491e5409895f3eaf63bdc05c6" member_type: VOTER last_known_addr { host: "127.10.198.2" port: 35073 } }
I20260925 22:44:56.001622 18487 leader_election.cc:288] T 8ade2068da5b404ea572164cf498c942 P 1ffb9543e6c340f78a7f82ffff34faf9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dcfd1608a17742d9a03fa549b12bb0c8 (127.10.198.1:41135), f5cc83f491e5409895f3eaf63bdc05c6 (127.10.198.2:35073)
I20260925 22:44:56.005687 18475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:44:56.002084 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.198.3:48215 (local address 127.10.198.1:41135)
0925 22:44:56.002208 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0925 22:44:56.002213 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:44:56.002230 (+ 17us) server_negotiation.cc:407] Connection header received
0925 22:44:56.002287 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:44:56.002290 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:44:56.002341 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:44:56.002425 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:44:56.002709 (+ 284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:44:56.003292 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:44:56.004557 (+ 1265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:44:56.005070 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:44:56.005208 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:44:56.005267 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0925 22:44:56.005468 (+ 201us) server_negotiation.cc:299] Negotiation successful
0925 22:44:56.005560 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260925 22:44:56.005709 18490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:44:56.001888 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.198.1:41135 (local address 127.10.198.3:48215)
0925 22:44:56.002041 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0925 22:44:56.002059 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0925 22:44:56.002191 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:44:56.002446 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:44:56.002449 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:44:56.002467 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:44:56.002585 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:44:56.002591 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:44:56.003418 (+ 827us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:44:56.003423 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:44:56.004407 (+ 984us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:44:56.004419 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:44:56.005205 (+ 786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:44:56.005209 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:44:56.005309 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:44:56.005348 (+ 39us) client_negotiation.cc:769] Sending connection context
0925 22:44:56.005428 (+ 80us) client_negotiation.cc:240] Negotiation successful
0925 22:44:56.005531 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":53,"mutex_wait_us":1}
I20260925 22:44:56.005793 18476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:44:56.002078 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.198.3:52669 (local address 127.10.198.2:35073)
0925 22:44:56.002225 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0925 22:44:56.002230 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:44:56.002413 (+ 183us) server_negotiation.cc:407] Connection header received
0925 22:44:56.002566 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:44:56.002569 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:44:56.002625 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:44:56.002709 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:44:56.002963 (+ 254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:44:56.003500 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:44:56.004676 (+ 1176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:44:56.005178 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:44:56.005295 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:44:56.005356 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0925 22:44:56.005576 (+ 220us) server_negotiation.cc:299] Negotiation successful
0925 22:44:56.005660 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260925 22:44:56.006239 18489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:44:56.001888 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.198.2:35073 (local address 127.10.198.3:52669)
0925 22:44:56.002325 (+ 437us) negotiation.cc:107] Waiting for socket to connect
0925 22:44:56.002336 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 22:44:56.002453 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:44:56.002720 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:44:56.002722 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:44:56.002738 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:44:56.002843 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:44:56.002857 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:44:56.003622 (+ 765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:44:56.003627 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:44:56.004547 (+ 920us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:44:56.004556 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:44:56.005338 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:44:56.005342 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:44:56.005441 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:44:56.005458 (+ 17us) client_negotiation.cc:769] Sending connection context
0925 22:44:56.006024 (+ 566us) client_negotiation.cc:240] Negotiation successful
0925 22:44:56.006104 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":370,"mutex_wait_us":267}
I20260925 22:44:56.006704 18201 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ade2068da5b404ea572164cf498c942" candidate_uuid: "1ffb9543e6c340f78a7f82ffff34faf9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f5cc83f491e5409895f3eaf63bdc05c6" is_pre_election: true
I20260925 22:44:56.006735 18064 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ade2068da5b404ea572164cf498c942" candidate_uuid: "1ffb9543e6c340f78a7f82ffff34faf9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dcfd1608a17742d9a03fa549b12bb0c8" is_pre_election: true
I20260925 22:44:56.006883 18201 raft_consensus.cc:2454] T 8ade2068da5b404ea572164cf498c942 P f5cc83f491e5409895f3eaf63bdc05c6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1ffb9543e6c340f78a7f82ffff34faf9 in term 0.
I20260925 22:44:56.006981 18064 raft_consensus.cc:2454] T 8ade2068da5b404ea572164cf498c942 P dcfd1608a17742d9a03fa549b12bb0c8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1ffb9543e6c340f78a7f82ffff34faf9 in term 0.
I20260925 22:44:56.007306 18272 leader_election.cc:302] T 8ade2068da5b404ea572164cf498c942 P 1ffb9543e6c340f78a7f82ffff34faf9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1ffb9543e6c340f78a7f82ffff34faf9, f5cc83f491e5409895f3eaf63bdc05c6; no voters:
I20260925 22:44:56.007620 18487 raft_consensus.cc:2781] T 8ade2068da5b404ea572164cf498c942 P 1ffb9543e6c340f78a7f82ffff34faf9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 22:44:56.007694 18487 raft_consensus.cc:482] T 8ade2068da5b404ea572164cf498c942 P 1ffb9543e6c340f78a7f82ffff34faf9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 22:44:56.007732 18487 raft_consensus.cc:3037] T 8ade2068da5b404ea572164cf498c942 P 1ffb9543e6c340f78a7f82ffff34faf9 [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:44:56.008844 18487 raft_consensus.cc:504] T 8ade2068da5b404ea572164cf498c942 P 1ffb9543e6c340f78a7f82ffff34faf9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ffb9543e6c340f78a7f82ffff34faf9" member_type: VOTER last_known_addr { host: "127.10.198.3" port: 46715 } } peers { permanent_uuid: "dcfd1608a17742d9a03fa549b12bb0c8" member_type: VOTER last_known_addr { host: "127.10.198.1" port: 41135 } } peers { permanent_uuid: "f5cc83f491e5409895f3eaf63bdc05c6" member_type: VOTER last_known_addr { host: "127.10.198.2" port: 35073 } }
I20260925 22:44:56.009251 18487 leader_election.cc:288] T 8ade2068da5b404ea572164cf498c942 P 1ffb9543e6c340f78a7f82ffff34faf9 [CANDIDATE]: Term 1 election: Requested vote from peers dcfd1608a17742d9a03fa549b12bb0c8 (127.10.198.1:41135), f5cc83f491e5409895f3eaf63bdc05c6 (127.10.198.2:35073)
I20260925 22:44:56.009708 18064 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ade2068da5b404ea572164cf498c942" candidate_uuid: "1ffb9543e6c340f78a7f82ffff34faf9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dcfd1608a17742d9a03fa549b12bb0c8"
I20260925 22:44:56.009709 18201 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ade2068da5b404ea572164cf498c942" candidate_uuid: "1ffb9543e6c340f78a7f82ffff34faf9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f5cc83f491e5409895f3eaf63bdc05c6"
I20260925 22:44:56.009868 18064 raft_consensus.cc:3037] T 8ade2068da5b404ea572164cf498c942 P dcfd1608a17742d9a03fa549b12bb0c8 [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:44:56.009876 18201 raft_consensus.cc:3037] T 8ade2068da5b404ea572164cf498c942 P f5cc83f491e5409895f3eaf63bdc05c6 [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:44:56.010855 18201 raft_consensus.cc:2454] T 8ade2068da5b404ea572164cf498c942 P f5cc83f491e5409895f3eaf63bdc05c6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1ffb9543e6c340f78a7f82ffff34faf9 in term 1.
I20260925 22:44:56.010947 18064 raft_consensus.cc:2454] T 8ade2068da5b404ea572164cf498c942 P dcfd1608a17742d9a03fa549b12bb0c8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1ffb9543e6c340f78a7f82ffff34faf9 in term 1.
I20260925 22:44:56.011337 18273 leader_election.cc:302] T 8ade2068da5b404ea572164cf498c942 P 1ffb9543e6c340f78a7f82ffff34faf9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1ffb9543e6c340f78a7f82ffff34faf9, dcfd1608a17742d9a03fa549b12bb0c8; no voters:
I20260925 22:44:56.011565 18487 raft_consensus.cc:2781] T 8ade2068da5b404ea572164cf498c942 P 1ffb9543e6c340f78a7f82ffff34faf9 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 22:44:56.011817 18487 raft_consensus.cc:686] T 8ade2068da5b404ea572164cf498c942 P 1ffb9543e6c340f78a7f82ffff34faf9 [term 1 LEADER]: Becoming Leader. State: Replica: 1ffb9543e6c340f78a7f82ffff34faf9, State: Running, Role: LEADER
I20260925 22:44:56.012130 18487 consensus_queue.cc:237] T 8ade2068da5b404ea572164cf498c942 P 1ffb9543e6c340f78a7f82ffff34faf9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ffb9543e6c340f78a7f82ffff34faf9" member_type: VOTER last_known_addr { host: "127.10.198.3" port: 46715 } } peers { permanent_uuid: "dcfd1608a17742d9a03fa549b12bb0c8" member_type: VOTER last_known_addr { host: "127.10.198.1" port: 41135 } } peers { permanent_uuid: "f5cc83f491e5409895f3eaf63bdc05c6" member_type: VOTER last_known_addr { host: "127.10.198.2" port: 35073 } }
I20260925 22:44:56.012691 18489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:44:56.012397 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.198.62:41969 (local address 127.10.198.3:47695)
0925 22:44:56.012540 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0925 22:44:56.012606 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.198.62:41969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260925 22:44:56.015002 17748 catalog_manager.cc:5495] T 8ade2068da5b404ea572164cf498c942 P 1ffb9543e6c340f78a7f82ffff34faf9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1ffb9543e6c340f78a7f82ffff34faf9 (127.10.198.3). New cstate: current_term: 1 leader_uuid: "1ffb9543e6c340f78a7f82ffff34faf9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ffb9543e6c340f78a7f82ffff34faf9" member_type: VOTER last_known_addr { host: "127.10.198.3" port: 46715 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dcfd1608a17742d9a03fa549b12bb0c8" member_type: VOTER last_known_addr { host: "127.10.198.1" port: 41135 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f5cc83f491e5409895f3eaf63bdc05c6" member_type: VOTER last_known_addr { host: "127.10.198.2" port: 35073 } health_report { overall_health: UNKNOWN } } }
I20260925 22:44:56.038287 18473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:44:56.034702 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44716 (local address 127.10.198.3:46715)
0925 22:44:56.034833 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0925 22:44:56.034837 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:44:56.034853 (+ 16us) server_negotiation.cc:407] Connection header received
0925 22:44:56.035004 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:44:56.035007 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:44:56.035066 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:44:56.035206 (+ 140us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 22:44:56.035444 (+ 238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:44:56.036034 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:44:56.036881 (+ 847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:44:56.037089 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:44:56.037169 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:44:56.037469 (+ 300us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 22:44:56.037949 (+ 480us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 22:44:56.038041 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0925 22:44:56.038112 (+ 71us) server_negotiation.cc:299] Negotiation successful
0925 22:44:56.038165 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260925 22:44:56.041515 18201 raft_consensus.cc:1261] T 8ade2068da5b404ea572164cf498c942 P f5cc83f491e5409895f3eaf63bdc05c6 [term 1 FOLLOWER]: Refusing update from remote peer 1ffb9543e6c340f78a7f82ffff34faf9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 22:44:56.041738 18064 raft_consensus.cc:1261] T 8ade2068da5b404ea572164cf498c942 P dcfd1608a17742d9a03fa549b12bb0c8 [term 1 FOLLOWER]: Refusing update from remote peer 1ffb9543e6c340f78a7f82ffff34faf9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 22:44:56.042177 18491 consensus_queue.cc:1035] T 8ade2068da5b404ea572164cf498c942 P 1ffb9543e6c340f78a7f82ffff34faf9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f5cc83f491e5409895f3eaf63bdc05c6" member_type: VOTER last_known_addr { host: "127.10.198.2" port: 35073 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 22:44:56.042435 18487 consensus_queue.cc:1035] T 8ade2068da5b404ea572164cf498c942 P 1ffb9543e6c340f78a7f82ffff34faf9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dcfd1608a17742d9a03fa549b12bb0c8" member_type: VOTER last_known_addr { host: "127.10.198.1" port: 41135 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 22:44:56.044476 18488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:44:56.044073 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.198.62:41969 (local address 127.10.198.2:45591)
0925 22:44:56.044269 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0925 22:44:56.044353 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.198.62:41969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120}
I20260925 22:44:56.046229 18486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:44:56.045041 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.198.62:41969 (local address 127.10.198.1:35695)
0925 22:44:56.045457 (+ 416us) negotiation.cc:107] Waiting for socket to connect
0925 22:44:56.045525 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.198.62:41969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":325}
I20260925 22:44:56.047708 18489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:44:56.047389 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.198.62:41969 (local address 127.10.198.3:37171)
0925 22:44:56.047544 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0925 22:44:56.047619 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.198.62:41969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260925 22:44:56.048597 18494 mvcc.cc:205] Tried to move back new op lower bound from 7333381308580425728 to 7333381308469104640. Current Snapshot: MvccSnapshot[applied={T|T < 7333381308580425728}]
I20260925 22:44:56.049721 18486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:44:56.049427 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.198.62:41969 (local address 127.10.198.1:33135)
0925 22:44:56.049545 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0925 22:44:56.049608 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.198.62:41969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20260925 22:44:56.050815 18488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:44:56.050512 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.198.62:41969 (local address 127.10.198.2:51249)
0925 22:44:56.050674 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0925 22:44:56.050732 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.198.62:41969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":86}
I20260925 22:44:56.051916 18489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:44:56.051630 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.198.62:41969 (local address 127.10.198.3:50145)
0925 22:44:56.051740 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0925 22:44:56.051816 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.198.62:41969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
I20260925 22:44:56.060397 17747 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59322:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 22:44:56.060578 17747 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59322:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 22:44:56.064939 17747 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f7d2a2767543443e8ba59b91bd97f4e1: Sending DeleteTablet for 3 replicas of tablet 8ade2068da5b404ea572164cf498c942
I20260925 22:44:56.065636 18318 tablet_service.cc:1514] Processing DeleteTablet for tablet 8ade2068da5b404ea572164cf498c942 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 22:44:56 UTC) from {username='oryx', principal='oryx/127.10.198.61@KRBTEST.COM'} at 127.0.0.1:44700
I20260925 22:44:56.065768 18044 tablet_service.cc:1514] Processing DeleteTablet for tablet 8ade2068da5b404ea572164cf498c942 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 22:44:56 UTC) from {username='oryx', principal='oryx/127.10.198.61@KRBTEST.COM'} at 127.0.0.1:48094
I20260925 22:44:56.066295 18504 tablet_replica.cc:330] T 8ade2068da5b404ea572164cf498c942 P dcfd1608a17742d9a03fa549b12bb0c8: stopping tablet replica
I20260925 22:44:56.066324 18181 tablet_service.cc:1514] Processing DeleteTablet for tablet 8ade2068da5b404ea572164cf498c942 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 22:44:56 UTC) from {username='oryx', principal='oryx/127.10.198.61@KRBTEST.COM'} at 127.0.0.1:42512
I20260925 22:44:56.066529 18504 raft_consensus.cc:2229] T 8ade2068da5b404ea572164cf498c942 P dcfd1608a17742d9a03fa549b12bb0c8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 22:44:56.066690 18505 tablet_replica.cc:330] T 8ade2068da5b404ea572164cf498c942 P f5cc83f491e5409895f3eaf63bdc05c6: stopping tablet replica
I20260925 22:44:56.066713 18504 raft_consensus.cc:2258] T 8ade2068da5b404ea572164cf498c942 P dcfd1608a17742d9a03fa549b12bb0c8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 22:44:56.066840 18505 raft_consensus.cc:2229] T 8ade2068da5b404ea572164cf498c942 P f5cc83f491e5409895f3eaf63bdc05c6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 22:44:56.066890 11032 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrWXm8X/build/debug/bin/kudu with pid 17892
I20260925 22:44:56.068917 18505 raft_consensus.cc:2258] T 8ade2068da5b404ea572164cf498c942 P f5cc83f491e5409895f3eaf63bdc05c6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 22:44:56.069727 18505 ts_tablet_manager.cc:1905] T 8ade2068da5b404ea572164cf498c942 P f5cc83f491e5409895f3eaf63bdc05c6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 22:44:56.072147 18505 ts_tablet_manager.cc:1918] T 8ade2068da5b404ea572164cf498c942 P f5cc83f491e5409895f3eaf63bdc05c6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 22:44:56.072266 18505 log.cc:1196] T 8ade2068da5b404ea572164cf498c942 P f5cc83f491e5409895f3eaf63bdc05c6: Deleting WAL directory at /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376259650642-11032-0/minicluster-data/ts-1/wal/wals/8ade2068da5b404ea572164cf498c942
I20260925 22:44:56.072544 18505 ts_tablet_manager.cc:1939] T 8ade2068da5b404ea572164cf498c942 P f5cc83f491e5409895f3eaf63bdc05c6: Deleting consensus metadata
I20260925 22:44:56.073385 17733 catalog_manager.cc:4841] TS f5cc83f491e5409895f3eaf63bdc05c6 (127.10.198.2:35073): tablet 8ade2068da5b404ea572164cf498c942 (table test-table [id=3aff0a87d5984c43848c1400a31a30a5]) successfully deleted
I20260925 22:44:56.076361 18503 tablet_replica.cc:330] T 8ade2068da5b404ea572164cf498c942 P 1ffb9543e6c340f78a7f82ffff34faf9: stopping tablet replica
I20260925 22:44:56.076582 18503 raft_consensus.cc:2229] T 8ade2068da5b404ea572164cf498c942 P 1ffb9543e6c340f78a7f82ffff34faf9 [term 1 LEADER]: Raft consensus shutting down.
I20260925 22:44:56.076889 11032 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrWXm8X/build/debug/bin/kudu with pid 18118
I20260925 22:44:56.077044 18474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:44:56.076716 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.198.1:41135 (local address 127.0.0.1:48110)
0925 22:44:56.076894 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0925 22:44:56.076953 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.198.1:41135: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":90}
W20260925 22:44:56.077232 17734 catalog_manager.cc:4584] TS dcfd1608a17742d9a03fa549b12bb0c8 (127.10.198.1:41135): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 8ade2068da5b404ea572164cf498c942: Network error: Client connection negotiation failed: client connection to 127.10.198.1:41135: connect: Connection refused (error 111)
I20260925 22:44:56.079241 18503 raft_consensus.cc:2258] T 8ade2068da5b404ea572164cf498c942 P 1ffb9543e6c340f78a7f82ffff34faf9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 22:44:56.080291 18503 ts_tablet_manager.cc:1905] T 8ade2068da5b404ea572164cf498c942 P 1ffb9543e6c340f78a7f82ffff34faf9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 22:44:56.085016 18503 ts_tablet_manager.cc:1918] T 8ade2068da5b404ea572164cf498c942 P 1ffb9543e6c340f78a7f82ffff34faf9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 22:44:56.085199 18503 log.cc:1196] T 8ade2068da5b404ea572164cf498c942 P 1ffb9543e6c340f78a7f82ffff34faf9: Deleting WAL directory at /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790376259650642-11032-0/minicluster-data/ts-2/wal/wals/8ade2068da5b404ea572164cf498c942
I20260925 22:44:56.085634 18503 ts_tablet_manager.cc:1939] T 8ade2068da5b404ea572164cf498c942 P 1ffb9543e6c340f78a7f82ffff34faf9: Deleting consensus metadata
I20260925 22:44:56.086289 11032 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrWXm8X/build/debug/bin/kudu with pid 18254
I20260925 22:44:56.094853 11032 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrWXm8X/build/debug/bin/kudu with pid 17611
I20260925 22:44:56.101482 11032 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrWXm8X/build/debug/bin/kudu with pid 17712
2026-09-25T22:44:56Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (5722 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790376259650642-11032-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 22:44:56 dist-test-slave-5vdr krb5kdc[18508](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 22:44:56 dist-test-slave-5vdr krb5kdc[18508](info): set up 2 sockets
Sep 25 22:44:56 dist-test-slave-5vdr krb5kdc[18508](info): commencing operation
krb5kdc: starting...
W20260925 22:44:57.160686 11032 mini_kdc.cc:121] Time spent starting KDC: real 1.034s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790376259650642-11032-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790376259650642-11032-0/krb5kdc/test-user.keytab.
Sep 25 22:44:57 dist-test-slave-5vdr krb5kdc[18508](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376297, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T22:44:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T22:44:57Z Disabled control of system clock
WARNING: no policy specified for oryx/127.10.198.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.10.198.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.10.198.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790376259650642-11032-0/krb5kdc/oryx_127.10.198.62.keytab.
Entry for principal oryx/127.10.198.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790376259650642-11032-0/krb5kdc/oryx_127.10.198.62.keytab.
I20260925 22:44:57.295930 11032 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrWXm8X/build/debug/bin/kudu
/tmp/dist-test-taskrWXm8X/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790376259650642-11032-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790376259650642-11032-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790376259650642-11032-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790376259650642-11032-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.198.62:46765
--webserver_interface=127.10.198.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790376259650642-11032-0/krb5kdc/oryx_127.10.198.62.keytab
--principal=oryx/127.10.198.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.198.20:36411
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.198.62:46765
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790376259650642-11032-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790376259650642-11032-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790376259650642-11032-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:44:57.419646 18521 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:44:57.420101 18521 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:44:57.420188 18521 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:44:57.424768 18521 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 22:44:57.424875 18521 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260925 22:44:57.424911 18521 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:44:57.424938 18521 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:44:57.424983 18521 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 22:44:57.425011 18521 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 22:44:57.434779 18521 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.198.20:36411
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790376259650642-11032-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790376259650642-11032-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.198.62:46765
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790376259650642-11032-0/krb5kdc/oryx_127.10.198.62.keytab
--principal=oryx/127.10.198.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.198.62:46765
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790376259650642-11032-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.198.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18521
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790376259650642-11032-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:09:16 UTC on ccycloud.quasar-afmhyw.root.comops.site
build id 4673379
I20260925 22:44:57.436067 18521 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:44:57.437117 18521 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:44:57.446520 18526 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:44:57.446467 18529 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:44:57.446837 18521 server_base.cc:1034] running on GCE node
W20260925 22:44:57.446470 18527 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:44:57.447623 18521 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:44:57.449105 18521 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:44:57.450362 18521 hybrid_clock.cc:648] HybridClock initialized: now 1790376297450317 us; error 65 us; skew 500 ppm
Sep 25 22:44:57 dist-test-slave-5vdr krb5kdc[18508](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376297, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.198.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:44:57.453744 18521 init.cc:377] Logged in from keytab as oryx/127.10.198.62@KRBTEST.COM (short username oryx)
I20260925 22:44:57.454849 18521 webserver.cc:460] Webserver started at http://127.10.198.62:40795/ using document root <none> and password file <none>
I20260925 22:44:57.455446 18521 fs_manager.cc:359] Metadata directory not provided
I20260925 22:44:57.455529 18521 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:44:57.455719 18521 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:44:57.458067 18521 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790376259650642-11032-0/minicluster-data/master-0/data/instance:
uuid: "9a42cb7f7adf47f1b0dd00bf3bfbd491"
format_stamp: "Formatted at 2026-09-25 22:44:57 on dist-test-slave-5vdr"
server_key: "7bc234ed173b4b180cd112e2e3c9b77e"
server_key_iv: "da32df8524b78bd89b29399854ea86ad"
server_key_version: "encryptionkey@0"
I20260925 22:44:57.458665 18521 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790376259650642-11032-0/minicluster-data/master-0/wal/instance:
uuid: "9a42cb7f7adf47f1b0dd00bf3bfbd491"
format_stamp: "Formatted at 2026-09-25 22:44:57 on dist-test-slave-5vdr"
server_key: "7bc234ed173b4b180cd112e2e3c9b77e"
server_key_iv: "da32df8524b78bd89b29399854ea86ad"
server_key_version: "encryptionkey@0"
I20260925 22:44:57.463007 18521 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260925 22:44:57.465958 18536 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:44:57.467521 18521 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260925 22:44:57.467743 18521 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790376259650642-11032-0/minicluster-data/master-0/data,/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790376259650642-11032-0/minicluster-data/master-0/wal
uuid: "9a42cb7f7adf47f1b0dd00bf3bfbd491"
format_stamp: "Formatted at 2026-09-25 22:44:57 on dist-test-slave-5vdr"
server_key: "7bc234ed173b4b180cd112e2e3c9b77e"
server_key_iv: "da32df8524b78bd89b29399854ea86ad"
server_key_version: "encryptionkey@0"
I20260925 22:44:57.467978 18521 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790376259650642-11032-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790376259650642-11032-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790376259650642-11032-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:44:57.496398 18521 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:44:57.500488 18521 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:44:57.500754 18521 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:44:57.510308 18521 rpc_server.cc:307] RPC server started. Bound to: 127.10.198.62:46765
I20260925 22:44:57.510334 18598 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.198.62:46765 every 8 connection(s)
I20260925 22:44:57.511531 18521 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790376259650642-11032-0/minicluster-data/master-0/data/info.pb
I20260925 22:44:57.513839 11032 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrWXm8X/build/debug/bin/kudu as pid 18521
I20260925 22:44:57.514029 11032 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790376259650642-11032-0/minicluster-data/master-0/wal/instance
I20260925 22:44:57.514395 11032 external_mini_cluster.cc:1428] Setting key 51e81ec73d11613226fb38c8c9e39d54
I20260925 22:44:57.516063 18599 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:44:57.524501 18599 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9a42cb7f7adf47f1b0dd00bf3bfbd491: Bootstrap starting.
Sep 25 22:44:57 dist-test-slave-5vdr krb5kdc[18508](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376297, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.10.198.62@KRBTEST.COM
I20260925 22:44:57.527899 18599 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9a42cb7f7adf47f1b0dd00bf3bfbd491: Neither blocks nor log segments found. Creating new log.
I20260925 22:44:57.529160 18599 log.cc:824] T 00000000000000000000000000000000 P 9a42cb7f7adf47f1b0dd00bf3bfbd491: Log is configured to *not* fsync() on all Append() calls
I20260925 22:44:57.531761 18599 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9a42cb7f7adf47f1b0dd00bf3bfbd491: No bootstrap required, opened a new log
I20260925 22:44:57.532546 18602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:44:57.516433 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55520 (local address 127.10.198.62:46765)
0925 22:44:57.516999 (+ 566us) server_negotiation.cc:206] Beginning negotiation
0925 22:44:57.517124 (+ 125us) server_negotiation.cc:399] Waiting for connection header
0925 22:44:57.517175 (+ 51us) server_negotiation.cc:407] Connection header received
0925 22:44:57.517890 (+ 715us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:44:57.517910 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:44:57.518311 (+ 401us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:44:57.518671 (+ 360us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:44:57.519731 (+ 1060us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:44:57.521527 (+ 1796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:44:57.522887 (+ 1360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:44:57.523473 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:44:57.523646 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:44:57.526806 (+ 3160us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:44:57.526812 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:44:57.526827 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:44:57.526866 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:44:57.528901 (+ 2035us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:44:57.529514 (+ 613us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:44:57.529521 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:44:57.529535 (+ 14us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:44:57.529726 (+ 191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:44:57.530110 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:44:57.530115 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:44:57.530117 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:44:57.530582 (+ 465us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:44:57.530746 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0925 22:44:57.531211 (+ 465us) server_negotiation.cc:299] Negotiation successful
0925 22:44:57.531588 (+ 377us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":394,"thread_start_us":204,"threads_started":1}
I20260925 22:44:57.535616 18599 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9a42cb7f7adf47f1b0dd00bf3bfbd491 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a42cb7f7adf47f1b0dd00bf3bfbd491" member_type: VOTER last_known_addr { host: "127.10.198.62" port: 46765 } }
I20260925 22:44:57.535970 18599 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9a42cb7f7adf47f1b0dd00bf3bfbd491 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:44:57.536161 18599 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9a42cb7f7adf47f1b0dd00bf3bfbd491 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a42cb7f7adf47f1b0dd00bf3bfbd491, State: Initialized, Role: FOLLOWER
I20260925 22:44:57.536752 18599 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9a42cb7f7adf47f1b0dd00bf3bfbd491 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a42cb7f7adf47f1b0dd00bf3bfbd491" member_type: VOTER last_known_addr { host: "127.10.198.62" port: 46765 } }
I20260925 22:44:57.536952 18599 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9a42cb7f7adf47f1b0dd00bf3bfbd491 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 22:44:57.537093 18599 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9a42cb7f7adf47f1b0dd00bf3bfbd491 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 22:44:57.537242 18599 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9a42cb7f7adf47f1b0dd00bf3bfbd491 [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:44:57.538477 18599 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9a42cb7f7adf47f1b0dd00bf3bfbd491 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a42cb7f7adf47f1b0dd00bf3bfbd491" member_type: VOTER last_known_addr { host: "127.10.198.62" port: 46765 } }
I20260925 22:44:57.538911 18599 leader_election.cc:302] T 00000000000000000000000000000000 P 9a42cb7f7adf47f1b0dd00bf3bfbd491 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9a42cb7f7adf47f1b0dd00bf3bfbd491; no voters:
I20260925 22:44:57.539299 18599 leader_election.cc:288] T 00000000000000000000000000000000 P 9a42cb7f7adf47f1b0dd00bf3bfbd491 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 22:44:57.539476 18604 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9a42cb7f7adf47f1b0dd00bf3bfbd491 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 22:44:57.539769 18604 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9a42cb7f7adf47f1b0dd00bf3bfbd491 [term 1 LEADER]: Becoming Leader. State: Replica: 9a42cb7f7adf47f1b0dd00bf3bfbd491, State: Running, Role: LEADER
I20260925 22:44:57.540161 18604 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9a42cb7f7adf47f1b0dd00bf3bfbd491 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a42cb7f7adf47f1b0dd00bf3bfbd491" member_type: VOTER last_known_addr { host: "127.10.198.62" port: 46765 } }
I20260925 22:44:57.540724 18599 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9a42cb7f7adf47f1b0dd00bf3bfbd491 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 22:44:57.541989 18606 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9a42cb7f7adf47f1b0dd00bf3bfbd491 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9a42cb7f7adf47f1b0dd00bf3bfbd491. Latest consensus state: current_term: 1 leader_uuid: "9a42cb7f7adf47f1b0dd00bf3bfbd491" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a42cb7f7adf47f1b0dd00bf3bfbd491" member_type: VOTER last_known_addr { host: "127.10.198.62" port: 46765 } } }
I20260925 22:44:57.542153 18606 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9a42cb7f7adf47f1b0dd00bf3bfbd491 [sys.catalog]: This master's current role is: LEADER
I20260925 22:44:57.542469 18605 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9a42cb7f7adf47f1b0dd00bf3bfbd491 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9a42cb7f7adf47f1b0dd00bf3bfbd491" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a42cb7f7adf47f1b0dd00bf3bfbd491" member_type: VOTER last_known_addr { host: "127.10.198.62" port: 46765 } } }
I20260925 22:44:57.542589 18605 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9a42cb7f7adf47f1b0dd00bf3bfbd491 [sys.catalog]: This master's current role is: LEADER
I20260925 22:44:57.542711 18613 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 22:44:57.546438 18613 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 22:44:57.553936 18613 catalog_manager.cc:1347] Generated new cluster ID: d9c09467ae4a4e1ba1c3888140c073dd
I20260925 22:44:57.554225 18613 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 22:44:57.569731 18613 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 22:44:57.570818 18613 catalog_manager.cc:1504] Loading token signing keys...
I20260925 22:44:57.583472 18613 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9a42cb7f7adf47f1b0dd00bf3bfbd491: Generated new TSK 0
I20260925 22:44:57.584569 18613 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.10.198.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.10.198.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.10.198.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790376259650642-11032-0/krb5kdc/oryx_127.10.198.1.keytab.
Entry for principal oryx/127.10.198.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790376259650642-11032-0/krb5kdc/oryx_127.10.198.1.keytab.
I20260925 22:44:57.624454 11032 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrWXm8X/build/debug/bin/kudu
/tmp/dist-test-taskrWXm8X/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790376259650642-11032-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790376259650642-11032-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790376259650642-11032-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790376259650642-11032-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.198.1:0
--local_ip_for_outbound_sockets=127.10.198.1
--webserver_interface=127.10.198.1
--webserver_port=0
--tserver_master_addrs=127.10.198.62:46765
--keytab_file=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790376259650642-11032-0/krb5kdc/oryx_127.10.198.1.keytab
--principal=oryx/127.10.198.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.198.20:36411
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790376259650642-11032-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790376259650642-11032-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790376259650642-11032-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:44:57.748903 18625 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:44:57.749226 18625 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:44:57.749275 18625 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:44:57.753948 18625 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:44:57.754033 18625 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:44:57.754076 18625 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260925 22:44:57.754163 18625 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.198.1
I20260925 22:44:57.760100 18625 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.198.20:36411
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790376259650642-11032-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790376259650642-11032-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790376259650642-11032-0/krb5kdc/oryx_127.10.198.1.keytab
--principal=oryx/127.10.198.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.198.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790376259650642-11032-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.198.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.10.198.62:46765
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18625
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.198.1
--log_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790376259650642-11032-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:09:16 UTC on ccycloud.quasar-afmhyw.root.comops.site
build id 4673379
I20260925 22:44:57.761423 18625 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:44:57.762583 18625 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:44:57.773576 18633 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:44:57.773576 18630 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:44:57.773607 18631 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:44:57.774318 18625 server_base.cc:1034] running on GCE node
I20260925 22:44:57.774812 18625 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:44:57.775525 18625 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:44:57.776728 18625 hybrid_clock.cc:648] HybridClock initialized: now 1790376297776695 us; error 47 us; skew 500 ppm
Sep 25 22:44:57 dist-test-slave-5vdr krb5kdc[18508](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376297, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.198.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:44:57.779946 18625 init.cc:377] Logged in from keytab as oryx/127.10.198.1@KRBTEST.COM (short username oryx)
I20260925 22:44:57.781077 18625 webserver.cc:460] Webserver started at http://127.10.198.1:36025/ using document root <none> and password file <none>
I20260925 22:44:57.781742 18625 fs_manager.cc:359] Metadata directory not provided
I20260925 22:44:57.781854 18625 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:44:57.782099 18625 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:44:57.784528 18625 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790376259650642-11032-0/minicluster-data/ts-0/data/instance:
uuid: "4e7aa6d387e94dffaff030eb416c66c4"
format_stamp: "Formatted at 2026-09-25 22:44:57 on dist-test-slave-5vdr"
server_key: "8dba6f191f4a44d3db1e20d16643ea91"
server_key_iv: "761429668cc2a72cfce55a986ca31803"
server_key_version: "encryptionkey@0"
I20260925 22:44:57.785215 18625 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790376259650642-11032-0/minicluster-data/ts-0/wal/instance:
uuid: "4e7aa6d387e94dffaff030eb416c66c4"
format_stamp: "Formatted at 2026-09-25 22:44:57 on dist-test-slave-5vdr"
server_key: "8dba6f191f4a44d3db1e20d16643ea91"
server_key_iv: "761429668cc2a72cfce55a986ca31803"
server_key_version: "encryptionkey@0"
I20260925 22:44:57.789608 18625 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260925 22:44:57.792649 18640 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:44:57.794238 18625 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 22:44:57.794440 18625 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790376259650642-11032-0/minicluster-data/ts-0/data,/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790376259650642-11032-0/minicluster-data/ts-0/wal
uuid: "4e7aa6d387e94dffaff030eb416c66c4"
format_stamp: "Formatted at 2026-09-25 22:44:57 on dist-test-slave-5vdr"
server_key: "8dba6f191f4a44d3db1e20d16643ea91"
server_key_iv: "761429668cc2a72cfce55a986ca31803"
server_key_version: "encryptionkey@0"
I20260925 22:44:57.794595 18625 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790376259650642-11032-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790376259650642-11032-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790376259650642-11032-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:44:57.811822 18625 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:44:57.815562 18625 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:44:57.815816 18625 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:44:57.817476 18625 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:44:57.817577 18625 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:44:57.817649 18625 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:44:57.817705 18625 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 22:44:57.830243 18625 rpc_server.cc:307] RPC server started. Bound to: 127.10.198.1:41409
I20260925 22:44:57.830374 18755 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.198.1:41409 every 8 connection(s)
Sep 25 22:44:57 dist-test-slave-5vdr krb5kdc[18508](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376297, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.198.1@KRBTEST.COM for oryx/127.10.198.62@KRBTEST.COM
I20260925 22:44:57.831398 18625 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790376259650642-11032-0/minicluster-data/ts-0/data/info.pb
I20260925 22:44:57.832338 11032 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrWXm8X/build/debug/bin/kudu as pid 18625
I20260925 22:44:57.832489 11032 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790376259650642-11032-0/minicluster-data/ts-0/wal/instance
I20260925 22:44:57.832842 11032 external_mini_cluster.cc:1428] Setting key a790453335606ef9f1340afb4c69c0bb
I20260925 22:44:57.836112 18602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:44:57.818735 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.198.1:51451 (local address 127.10.198.62:46765)
0925 22:44:57.818907 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0925 22:44:57.818921 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0925 22:44:57.820004 (+ 1083us) server_negotiation.cc:407] Connection header received
0925 22:44:57.821229 (+ 1225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:44:57.821233 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:44:57.821306 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:44:57.821441 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:44:57.823496 (+ 2055us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:44:57.824476 (+ 980us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:44:57.827119 (+ 2643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:44:57.827555 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:44:57.827712 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:44:57.831225 (+ 3513us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:44:57.831231 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:44:57.831234 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:44:57.831269 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:44:57.833265 (+ 1996us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:44:57.833923 (+ 658us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:44:57.833927 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:44:57.833929 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:44:57.834035 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:44:57.834685 (+ 650us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:44:57.834688 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:44:57.834689 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:44:57.834885 (+ 196us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:44:57.834972 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0925 22:44:57.835767 (+ 795us) server_negotiation.cc:299] Negotiation successful
0925 22:44:57.835974 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260925 22:44:57.837186 18649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:44:57.819219 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.198.62:46765 (local address 127.10.198.1:51451)
0925 22:44:57.819749 (+ 530us) negotiation.cc:107] Waiting for socket to connect
0925 22:44:57.819808 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0925 22:44:57.820971 (+ 1163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:44:57.821718 (+ 747us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:44:57.821728 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:44:57.822334 (+ 606us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:44:57.823222 (+ 888us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:44:57.823254 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:44:57.824705 (+ 1451us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:44:57.824709 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:44:57.826904 (+ 2195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:44:57.826927 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:44:57.827801 (+ 874us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:44:57.827809 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:44:57.828127 (+ 318us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:44:57.828999 (+ 872us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:44:57.829039 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:44:57.830970 (+ 1931us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:44:57.833454 (+ 2484us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:44:57.833461 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:44:57.833479 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:44:57.833807 (+ 328us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:44:57.834278 (+ 471us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:44:57.834281 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:44:57.834282 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:44:57.834562 (+ 280us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:44:57.835011 (+ 449us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:44:57.835022 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:44:57.835461 (+ 439us) client_negotiation.cc:769] Sending connection context
0925 22:44:57.835727 (+ 266us) client_negotiation.cc:240] Negotiation successful
0925 22:44:57.836196 (+ 469us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":301,"thread_start_us":157,"threads_started":1}
I20260925 22:44:57.839033 18756 heartbeater.cc:347] Connected to a master server at 127.10.198.62:46765
I20260925 22:44:57.839236 18756 heartbeater.cc:464] Registering TS with master...
I20260925 22:44:57.839912 18756 heartbeater.cc:511] Master 127.10.198.62:46765 requested a full tablet report, sending...
I20260925 22:44:57.842056 18552 ts_manager.cc:194] Registered new tserver with Master: 4e7aa6d387e94dffaff030eb416c66c4 (127.10.198.1:41409)
I20260925 22:44:57.843657 18552 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.10.198.1@KRBTEST.COM'} at 127.10.198.1:51451
WARNING: no policy specified for oryx/127.10.198.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.10.198.2@KRBTEST.COM" created.
I20260925 22:44:57.851249 18602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:44:57.841791 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.198.1:43181 (local address 127.10.198.62:46765)
0925 22:44:57.841934 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0925 22:44:57.841939 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:44:57.842323 (+ 384us) server_negotiation.cc:407] Connection header received
0925 22:44:57.842496 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:44:57.842500 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:44:57.842620 (+ 120us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:44:57.842710 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:44:57.843344 (+ 634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:44:57.843983 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:44:57.845179 (+ 1196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:44:57.845590 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:44:57.845720 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:44:57.847216 (+ 1496us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:44:57.847223 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:44:57.847229 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:44:57.847265 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:44:57.849034 (+ 1769us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:44:57.849650 (+ 616us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:44:57.849654 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:44:57.849655 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:44:57.849781 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:44:57.850149 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:44:57.850155 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:44:57.850158 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:44:57.850513 (+ 355us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:44:57.850633 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0925 22:44:57.850938 (+ 305us) server_negotiation.cc:299] Negotiation successful
0925 22:44:57.851067 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260925 22:44:57.851276 18763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:44:57.841778 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.198.62:46765 (local address 127.10.198.1:43181)
0925 22:44:57.842217 (+ 439us) negotiation.cc:107] Waiting for socket to connect
0925 22:44:57.842237 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0925 22:44:57.842378 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:44:57.842771 (+ 393us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:44:57.842775 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:44:57.843046 (+ 271us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:44:57.843201 (+ 155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:44:57.843211 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:44:57.844146 (+ 935us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:44:57.844150 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:44:57.845019 (+ 869us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:44:57.845032 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:44:57.845755 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:44:57.845759 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:44:57.845859 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:44:57.846552 (+ 693us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:44:57.846573 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:44:57.847015 (+ 442us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:44:57.849190 (+ 2175us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:44:57.849193 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:44:57.849195 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:44:57.849508 (+ 313us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:44:57.849893 (+ 385us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:44:57.849896 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:44:57.849897 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:44:57.850030 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:44:57.850639 (+ 609us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:44:57.850642 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:44:57.850747 (+ 105us) client_negotiation.cc:769] Sending connection context
0925 22:44:57.850874 (+ 127us) client_negotiation.cc:240] Negotiation successful
0925 22:44:57.851059 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":325,"thread_start_us":216,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.10.198.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790376259650642-11032-0/krb5kdc/oryx_127.10.198.2.keytab.
Entry for principal oryx/127.10.198.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790376259650642-11032-0/krb5kdc/oryx_127.10.198.2.keytab.
I20260925 22:44:57.869566 11032 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrWXm8X/build/debug/bin/kudu
/tmp/dist-test-taskrWXm8X/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790376259650642-11032-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790376259650642-11032-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790376259650642-11032-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790376259650642-11032-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.198.2:0
--local_ip_for_outbound_sockets=127.10.198.2
--webserver_interface=127.10.198.2
--webserver_port=0
--tserver_master_addrs=127.10.198.62:46765
--keytab_file=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790376259650642-11032-0/krb5kdc/oryx_127.10.198.2.keytab
--principal=oryx/127.10.198.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.198.20:36411
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790376259650642-11032-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790376259650642-11032-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790376259650642-11032-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:44:57.991796 18765 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:44:57.992242 18765 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:44:57.992296 18765 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:44:57.996785 18765 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:44:57.996871 18765 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:44:57.996929 18765 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260925 22:44:57.997012 18765 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.198.2
I20260925 22:44:58.003219 18765 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.198.20:36411
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790376259650642-11032-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790376259650642-11032-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790376259650642-11032-0/krb5kdc/oryx_127.10.198.2.keytab
--principal=oryx/127.10.198.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.198.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790376259650642-11032-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.198.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.10.198.62:46765
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18765
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.198.2
--log_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790376259650642-11032-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:09:16 UTC on ccycloud.quasar-afmhyw.root.comops.site
build id 4673379
I20260925 22:44:58.004544 18765 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:44:58.005726 18765 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:44:58.016871 18771 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:44:58.016820 18773 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:44:58.016820 18770 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:44:58.017177 18765 server_base.cc:1034] running on GCE node
I20260925 22:44:58.017758 18765 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:44:58.018553 18765 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:44:58.019790 18765 hybrid_clock.cc:648] HybridClock initialized: now 1790376298019766 us; error 45 us; skew 500 ppm
Sep 25 22:44:58 dist-test-slave-5vdr krb5kdc[18508](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376298, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.198.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:44:58.023327 18765 init.cc:377] Logged in from keytab as oryx/127.10.198.2@KRBTEST.COM (short username oryx)
I20260925 22:44:58.024466 18765 webserver.cc:460] Webserver started at http://127.10.198.2:34111/ using document root <none> and password file <none>
I20260925 22:44:58.025106 18765 fs_manager.cc:359] Metadata directory not provided
I20260925 22:44:58.025208 18765 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:44:58.025460 18765 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:44:58.027801 18765 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790376259650642-11032-0/minicluster-data/ts-1/data/instance:
uuid: "fe72d9fd358f4bf7967ac3ea2264aed9"
format_stamp: "Formatted at 2026-09-25 22:44:58 on dist-test-slave-5vdr"
server_key: "81efe01ebdf0f042ad52fe224ed3a74f"
server_key_iv: "3f6234be4c777212389cb80a409053b0"
server_key_version: "encryptionkey@0"
I20260925 22:44:58.028577 18765 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790376259650642-11032-0/minicluster-data/ts-1/wal/instance:
uuid: "fe72d9fd358f4bf7967ac3ea2264aed9"
format_stamp: "Formatted at 2026-09-25 22:44:58 on dist-test-slave-5vdr"
server_key: "81efe01ebdf0f042ad52fe224ed3a74f"
server_key_iv: "3f6234be4c777212389cb80a409053b0"
server_key_version: "encryptionkey@0"
I20260925 22:44:58.033342 18765 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260925 22:44:58.036374 18780 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:44:58.037951 18765 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 22:44:58.038254 18765 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790376259650642-11032-0/minicluster-data/ts-1/data,/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790376259650642-11032-0/minicluster-data/ts-1/wal
uuid: "fe72d9fd358f4bf7967ac3ea2264aed9"
format_stamp: "Formatted at 2026-09-25 22:44:58 on dist-test-slave-5vdr"
server_key: "81efe01ebdf0f042ad52fe224ed3a74f"
server_key_iv: "3f6234be4c777212389cb80a409053b0"
server_key_version: "encryptionkey@0"
I20260925 22:44:58.038455 18765 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790376259650642-11032-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790376259650642-11032-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790376259650642-11032-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:44:58.063230 18765 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:44:58.067066 18765 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:44:58.067325 18765 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:44:58.068930 18765 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:44:58.069023 18765 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:44:58.069092 18765 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:44:58.069144 18765 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 25 22:44:58 dist-test-slave-5vdr krb5kdc[18508](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376298, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.198.2@KRBTEST.COM for oryx/127.10.198.62@KRBTEST.COM
I20260925 22:44:58.081498 18765 rpc_server.cc:307] RPC server started. Bound to: 127.10.198.2:41061
I20260925 22:44:58.081569 18895 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.198.2:41061 every 8 connection(s)
I20260925 22:44:58.082715 18765 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790376259650642-11032-0/minicluster-data/ts-1/data/info.pb
I20260925 22:44:58.086956 18602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:44:58.070177 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.198.2:50601 (local address 127.10.198.62:46765)
0925 22:44:58.070338 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0925 22:44:58.070342 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:44:58.071616 (+ 1274us) server_negotiation.cc:407] Connection header received
0925 22:44:58.073184 (+ 1568us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:44:58.073191 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:44:58.073271 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:44:58.073379 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:44:58.075270 (+ 1891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:44:58.076130 (+ 860us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:44:58.077623 (+ 1493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:44:58.077937 (+ 314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:44:58.078031 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:44:58.081947 (+ 3916us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:44:58.081951 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:44:58.081954 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:44:58.081989 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:44:58.084117 (+ 2128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:44:58.084856 (+ 739us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:44:58.084859 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:44:58.084860 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:44:58.084970 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:44:58.085452 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:44:58.085456 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:44:58.085457 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:44:58.085746 (+ 289us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:44:58.085845 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0925 22:44:58.086655 (+ 810us) server_negotiation.cc:299] Negotiation successful
0925 22:44:58.086793 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260925 22:44:58.087908 11032 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrWXm8X/build/debug/bin/kudu as pid 18765
I20260925 22:44:58.088033 11032 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790376259650642-11032-0/minicluster-data/ts-1/wal/instance
I20260925 22:44:58.088120 18789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:44:58.070821 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.198.62:46765 (local address 127.10.198.2:50601)
0925 22:44:58.071397 (+ 576us) negotiation.cc:107] Waiting for socket to connect
0925 22:44:58.071453 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0925 22:44:58.072901 (+ 1448us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:44:58.073609 (+ 708us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:44:58.073616 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:44:58.074225 (+ 609us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:44:58.075065 (+ 840us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:44:58.075107 (+ 42us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:44:58.076282 (+ 1175us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:44:58.076286 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:44:58.077389 (+ 1103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:44:58.077427 (+ 38us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:44:58.078114 (+ 687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:44:58.078119 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:44:58.078374 (+ 255us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:44:58.079287 (+ 913us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:44:58.079315 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:44:58.081620 (+ 2305us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:44:58.084293 (+ 2673us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:44:58.084299 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:44:58.084314 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:44:58.084693 (+ 379us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:44:58.085122 (+ 429us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:44:58.085126 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:44:58.085128 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:44:58.085312 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:44:58.085919 (+ 607us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:44:58.085939 (+ 20us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:44:58.086348 (+ 409us) client_negotiation.cc:769] Sending connection context
0925 22:44:58.086562 (+ 214us) client_negotiation.cc:240] Negotiation successful
0925 22:44:58.086887 (+ 325us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":350,"thread_start_us":143,"threads_started":1}
I20260925 22:44:58.088387 11032 external_mini_cluster.cc:1428] Setting key abc5ca3497dada688778d40864f98d65
I20260925 22:44:58.089615 18896 heartbeater.cc:347] Connected to a master server at 127.10.198.62:46765
I20260925 22:44:58.089876 18896 heartbeater.cc:464] Registering TS with master...
I20260925 22:44:58.090705 18896 heartbeater.cc:511] Master 127.10.198.62:46765 requested a full tablet report, sending...
I20260925 22:44:58.092195 18552 ts_manager.cc:194] Registered new tserver with Master: fe72d9fd358f4bf7967ac3ea2264aed9 (127.10.198.2:41061)
I20260925 22:44:58.093101 18552 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.10.198.2@KRBTEST.COM'} at 127.10.198.2:50601
WARNING: no policy specified for oryx/127.10.198.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.10.198.3@KRBTEST.COM" created.
I20260925 22:44:58.103227 18602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:44:58.093496 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.198.2:32939 (local address 127.10.198.62:46765)
0925 22:44:58.093641 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0925 22:44:58.093645 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:44:58.093904 (+ 259us) server_negotiation.cc:407] Connection header received
0925 22:44:58.094068 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:44:58.094071 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:44:58.094122 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:44:58.094217 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:44:58.094877 (+ 660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:44:58.095617 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:44:58.097192 (+ 1575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:44:58.097431 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:44:58.097537 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:44:58.099059 (+ 1522us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:44:58.099062 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:44:58.099064 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:44:58.099094 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:44:58.101063 (+ 1969us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:44:58.101659 (+ 596us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:44:58.101662 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:44:58.101663 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:44:58.101751 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:44:58.102157 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:44:58.102164 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:44:58.102166 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:44:58.102433 (+ 267us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:44:58.102530 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0925 22:44:58.102912 (+ 382us) server_negotiation.cc:299] Negotiation successful
0925 22:44:58.103079 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260925 22:44:58.103487 18903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:44:58.092331 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.198.62:46765 (local address 127.10.198.2:32939)
0925 22:44:58.093804 (+ 1473us) negotiation.cc:107] Waiting for socket to connect
0925 22:44:58.093822 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0925 22:44:58.093935 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:44:58.094261 (+ 326us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:44:58.094267 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:44:58.094549 (+ 282us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:44:58.094715 (+ 166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:44:58.094727 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:44:58.095804 (+ 1077us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:44:58.095811 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:44:58.096932 (+ 1121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:44:58.096952 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:44:58.097591 (+ 639us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:44:58.097598 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:44:58.097727 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:44:58.098311 (+ 584us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:44:58.098328 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:44:58.098841 (+ 513us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:44:58.101221 (+ 2380us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:44:58.101225 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:44:58.101226 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:44:58.101546 (+ 320us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:44:58.101847 (+ 301us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:44:58.101849 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:44:58.101851 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:44:58.101934 (+ 83us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:44:58.102547 (+ 613us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:44:58.102549 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:44:58.102698 (+ 149us) client_negotiation.cc:769] Sending connection context
0925 22:44:58.102852 (+ 154us) client_negotiation.cc:240] Negotiation successful
0925 22:44:58.103167 (+ 315us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1369,"thread_start_us":137,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.10.198.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790376259650642-11032-0/krb5kdc/oryx_127.10.198.3.keytab.
Entry for principal oryx/127.10.198.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790376259650642-11032-0/krb5kdc/oryx_127.10.198.3.keytab.
I20260925 22:44:58.121109 11032 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrWXm8X/build/debug/bin/kudu
/tmp/dist-test-taskrWXm8X/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790376259650642-11032-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790376259650642-11032-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790376259650642-11032-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790376259650642-11032-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.198.3:0
--local_ip_for_outbound_sockets=127.10.198.3
--webserver_interface=127.10.198.3
--webserver_port=0
--tserver_master_addrs=127.10.198.62:46765
--keytab_file=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790376259650642-11032-0/krb5kdc/oryx_127.10.198.3.keytab
--principal=oryx/127.10.198.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.198.20:36411
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790376259650642-11032-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790376259650642-11032-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790376259650642-11032-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:44:58.246941 18905 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:44:58.247287 18905 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:44:58.247336 18905 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:44:58.252030 18905 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:44:58.252123 18905 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:44:58.252168 18905 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260925 22:44:58.252255 18905 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.198.3
I20260925 22:44:58.258389 18905 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.198.20:36411
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790376259650642-11032-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790376259650642-11032-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790376259650642-11032-0/krb5kdc/oryx_127.10.198.3.keytab
--principal=oryx/127.10.198.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.198.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790376259650642-11032-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.198.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.10.198.62:46765
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18905
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.198.3
--log_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790376259650642-11032-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:09:16 UTC on ccycloud.quasar-afmhyw.root.comops.site
build id 4673379
I20260925 22:44:58.259625 18905 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:44:58.260836 18905 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:44:58.272014 18911 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:44:58.272014 18913 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:44:58.272027 18910 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:44:58.272146 18905 server_base.cc:1034] running on GCE node
I20260925 22:44:58.272778 18905 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:44:58.273667 18905 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:44:58.274857 18905 hybrid_clock.cc:648] HybridClock initialized: now 1790376298274837 us; error 40 us; skew 500 ppm
Sep 25 22:44:58 dist-test-slave-5vdr krb5kdc[18508](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376298, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.198.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:44:58.278195 18905 init.cc:377] Logged in from keytab as oryx/127.10.198.3@KRBTEST.COM (short username oryx)
I20260925 22:44:58.279273 18905 webserver.cc:460] Webserver started at http://127.10.198.3:37645/ using document root <none> and password file <none>
I20260925 22:44:58.280000 18905 fs_manager.cc:359] Metadata directory not provided
I20260925 22:44:58.280081 18905 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:44:58.280318 18905 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:44:58.282788 18905 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790376259650642-11032-0/minicluster-data/ts-2/data/instance:
uuid: "7d80f46d2eac41e7a9f0b2b2e1972abd"
format_stamp: "Formatted at 2026-09-25 22:44:58 on dist-test-slave-5vdr"
server_key: "5d2bbeffdcde06984ba2ae1d0513b5a8"
server_key_iv: "e2a954243b25fc2c808a61be792994e8"
server_key_version: "encryptionkey@0"
I20260925 22:44:58.283468 18905 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790376259650642-11032-0/minicluster-data/ts-2/wal/instance:
uuid: "7d80f46d2eac41e7a9f0b2b2e1972abd"
format_stamp: "Formatted at 2026-09-25 22:44:58 on dist-test-slave-5vdr"
server_key: "5d2bbeffdcde06984ba2ae1d0513b5a8"
server_key_iv: "e2a954243b25fc2c808a61be792994e8"
server_key_version: "encryptionkey@0"
I20260925 22:44:58.288496 18905 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260925 22:44:58.291662 18920 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:44:58.293256 18905 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 22:44:58.293440 18905 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790376259650642-11032-0/minicluster-data/ts-2/data,/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790376259650642-11032-0/minicluster-data/ts-2/wal
uuid: "7d80f46d2eac41e7a9f0b2b2e1972abd"
format_stamp: "Formatted at 2026-09-25 22:44:58 on dist-test-slave-5vdr"
server_key: "5d2bbeffdcde06984ba2ae1d0513b5a8"
server_key_iv: "e2a954243b25fc2c808a61be792994e8"
server_key_version: "encryptionkey@0"
I20260925 22:44:58.293597 18905 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790376259650642-11032-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790376259650642-11032-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790376259650642-11032-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:44:58.306967 18905 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:44:58.310855 18905 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:44:58.311120 18905 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:44:58.312891 18905 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:44:58.312999 18905 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:44:58.313071 18905 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:44:58.313117 18905 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 25 22:44:58 dist-test-slave-5vdr krb5kdc[18508](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376298, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.198.3@KRBTEST.COM for oryx/127.10.198.62@KRBTEST.COM
I20260925 22:44:58.326742 18905 rpc_server.cc:307] RPC server started. Bound to: 127.10.198.3:37891
I20260925 22:44:58.326812 19035 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.198.3:37891 every 8 connection(s)
I20260925 22:44:58.327898 18905 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790376259650642-11032-0/minicluster-data/ts-2/data/info.pb
I20260925 22:44:58.329310 11032 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrWXm8X/build/debug/bin/kudu as pid 18905
I20260925 22:44:58.329471 11032 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790376259650642-11032-0/minicluster-data/ts-2/wal/instance
I20260925 22:44:58.329535 18602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:44:58.314099 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.198.3:34813 (local address 127.10.198.62:46765)
0925 22:44:58.314279 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0925 22:44:58.314285 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 22:44:58.315383 (+ 1098us) server_negotiation.cc:407] Connection header received
0925 22:44:58.316856 (+ 1473us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:44:58.316863 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:44:58.316978 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:44:58.317117 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:44:58.318742 (+ 1625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:44:58.319379 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:44:58.320853 (+ 1474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:44:58.321171 (+ 318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:44:58.321303 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:44:58.324478 (+ 3175us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:44:58.324483 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:44:58.324488 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:44:58.324529 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:44:58.326429 (+ 1900us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:44:58.327248 (+ 819us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:44:58.327252 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:44:58.327254 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:44:58.327358 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:44:58.327747 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:44:58.327751 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:44:58.327752 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:44:58.328081 (+ 329us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:44:58.328197 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0925 22:44:58.329148 (+ 951us) server_negotiation.cc:299] Negotiation successful
0925 22:44:58.329309 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260925 22:44:58.329833 11032 external_mini_cluster.cc:1428] Setting key 770194d5f6f42cb2618884372f399f82
I20260925 22:44:58.330214 18929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:44:58.314606 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.198.62:46765 (local address 127.10.198.3:34813)
0925 22:44:58.315163 (+ 557us) negotiation.cc:107] Waiting for socket to connect
0925 22:44:58.315224 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0925 22:44:58.316469 (+ 1245us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:44:58.317414 (+ 945us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:44:58.317429 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:44:58.317877 (+ 448us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:44:58.318541 (+ 664us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:44:58.318565 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:44:58.319530 (+ 965us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:44:58.319534 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:44:58.320679 (+ 1145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:44:58.320697 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:44:58.321365 (+ 668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:44:58.321369 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:44:58.321632 (+ 263us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:44:58.322455 (+ 823us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:44:58.322477 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:44:58.324248 (+ 1771us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:44:58.326575 (+ 2327us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:44:58.326584 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:44:58.326605 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:44:58.327116 (+ 511us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:44:58.327462 (+ 346us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:44:58.327467 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:44:58.327469 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:44:58.327633 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:44:58.328196 (+ 563us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:44:58.328205 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:44:58.328511 (+ 306us) client_negotiation.cc:769] Sending connection context
0925 22:44:58.328732 (+ 221us) client_negotiation.cc:240] Negotiation successful
0925 22:44:58.329139 (+ 407us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":320,"thread_start_us":162,"threads_started":1}
I20260925 22:44:58.331583 19036 heartbeater.cc:347] Connected to a master server at 127.10.198.62:46765
I20260925 22:44:58.331766 19036 heartbeater.cc:464] Registering TS with master...
I20260925 22:44:58.332402 19036 heartbeater.cc:511] Master 127.10.198.62:46765 requested a full tablet report, sending...
I20260925 22:44:58.333765 18552 ts_manager.cc:194] Registered new tserver with Master: 7d80f46d2eac41e7a9f0b2b2e1972abd (127.10.198.3:37891)
I20260925 22:44:58.334453 18552 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.10.198.3@KRBTEST.COM'} at 127.10.198.3:34813
I20260925 22:44:58.344483 18602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:44:58.334614 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.198.3:35541 (local address 127.10.198.62:46765)
0925 22:44:58.334781 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0925 22:44:58.334786 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:44:58.335200 (+ 414us) server_negotiation.cc:407] Connection header received
0925 22:44:58.335282 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:44:58.335285 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:44:58.335333 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:44:58.335421 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:44:58.335898 (+ 477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:44:58.336674 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:44:58.337536 (+ 862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:44:58.337761 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:44:58.337846 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:44:58.339498 (+ 1652us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:44:58.339506 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:44:58.339511 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:44:58.339554 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:44:58.342178 (+ 2624us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:44:58.342902 (+ 724us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:44:58.342908 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:44:58.342913 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:44:58.343025 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:44:58.343354 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:44:58.343356 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:44:58.343357 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:44:58.343630 (+ 273us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:44:58.343735 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0925 22:44:58.344176 (+ 441us) server_negotiation.cc:299] Negotiation successful
0925 22:44:58.344323 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260925 22:44:58.344585 19042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:44:58.334452 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.198.62:46765 (local address 127.10.198.3:35541)
0925 22:44:58.335091 (+ 639us) negotiation.cc:107] Waiting for socket to connect
0925 22:44:58.335107 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 22:44:58.335212 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:44:58.335444 (+ 232us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:44:58.335446 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:44:58.335645 (+ 199us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:44:58.335747 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:44:58.335753 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:44:58.336830 (+ 1077us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:44:58.336832 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:44:58.337417 (+ 585us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:44:58.337425 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:44:58.337892 (+ 467us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:44:58.337895 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:44:58.337969 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:44:58.338611 (+ 642us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:44:58.338628 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:44:58.339110 (+ 482us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:44:58.342338 (+ 3228us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:44:58.342341 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:44:58.342343 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:44:58.342686 (+ 343us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:44:58.343135 (+ 449us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:44:58.343138 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:44:58.343140 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:44:58.343239 (+ 99us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:44:58.343770 (+ 531us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:44:58.343777 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:44:58.343940 (+ 163us) client_negotiation.cc:769] Sending connection context
0925 22:44:58.344096 (+ 156us) client_negotiation.cc:240] Negotiation successful
0925 22:44:58.344308 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":533,"thread_start_us":413,"threads_started":1}
I20260925 22:44:58.345340 11032 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 25 22:44:58 dist-test-slave-5vdr krb5kdc[18508](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376298, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 25 22:44:58 dist-test-slave-5vdr krb5kdc[18508](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.10.198.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 22:44:58 dist-test-slave-5vdr krb5kdc[18508](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.10.198.62@KRBTEST.COM, Server not found in Kerberos database
W20260925 22:44:58.374148 18602 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:44:58.366140 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55528 (local address 127.10.198.62:46765)
0925 22:44:58.366306 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0925 22:44:58.366310 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:44:58.366322 (+ 12us) server_negotiation.cc:407] Connection header received
0925 22:44:58.366361 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:44:58.366364 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:44:58.366404 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:44:58.366503 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:44:58.367305 (+ 802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:44:58.368291 (+ 986us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:44:58.369336 (+ 1045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:44:58.369796 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:44:58.369934 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:44:58.373968 (+ 4034us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55528: BlockingRecv error: recv got EOF from 127.0.0.1:55528 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
Sep 25 22:44:58 dist-test-slave-5vdr krb5kdc[18508](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376298, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.10.198.62@KRBTEST.COM
I20260925 22:44:58.385285 18602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:44:58.375746 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55532 (local address 127.10.198.62:46765)
0925 22:44:58.375948 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0925 22:44:58.375958 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 22:44:58.376020 (+ 62us) server_negotiation.cc:407] Connection header received
0925 22:44:58.376176 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:44:58.376178 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:44:58.376223 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:44:58.376304 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:44:58.376971 (+ 667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:44:58.377501 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:44:58.378351 (+ 850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:44:58.378573 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:44:58.378660 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:44:58.381524 (+ 2864us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:44:58.381527 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:44:58.381530 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:44:58.381565 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:44:58.383243 (+ 1678us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:44:58.383919 (+ 676us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:44:58.383923 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:44:58.383925 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:44:58.384018 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:44:58.384343 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:44:58.384346 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:44:58.384349 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:44:58.384569 (+ 220us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:44:58.384720 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0925 22:44:58.384981 (+ 261us) server_negotiation.cc:299] Negotiation successful
0925 22:44:58.385112 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260925 22:44:58.389128 18552 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55532:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 22:44:58.392545 18552 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 22:44:58.410548 19063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:44:58.403742 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60530 (local address 127.10.198.3:37891)
0925 22:44:58.404076 (+ 334us) server_negotiation.cc:206] Beginning negotiation
0925 22:44:58.404089 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0925 22:44:58.404428 (+ 339us) server_negotiation.cc:407] Connection header received
0925 22:44:58.404539 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:44:58.404543 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:44:58.404765 (+ 222us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:44:58.405273 (+ 508us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:44:58.405418 (+ 145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:44:58.406583 (+ 1165us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:44:58.408616 (+ 2033us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:44:58.409502 (+ 886us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:44:58.409639 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:44:58.409832 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0925 22:44:58.410261 (+ 429us) server_negotiation.cc:299] Negotiation successful
0925 22:44:58.410377 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":231,"thread_start_us":144,"threads_started":1}
I20260925 22:44:58.410535 19061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:44:58.403194 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.198.3:37891 (local address 127.0.0.1:60530)
0925 22:44:58.404347 (+ 1153us) negotiation.cc:107] Waiting for socket to connect
0925 22:44:58.404359 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 22:44:58.404445 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:44:58.404923 (+ 478us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:44:58.404926 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:44:58.404951 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:44:58.405090 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:44:58.405110 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:44:58.406711 (+ 1601us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:44:58.406714 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:44:58.408426 (+ 1712us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:44:58.408440 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:44:58.409890 (+ 1450us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:44:58.409893 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:44:58.410041 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:44:58.410086 (+ 45us) client_negotiation.cc:769] Sending connection context
0925 22:44:58.410223 (+ 137us) client_negotiation.cc:240] Negotiation successful
0925 22:44:58.410377 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1103,"thread_start_us":91,"threads_started":1}
I20260925 22:44:58.411217 19059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:44:58.402757 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.198.2:41061 (local address 127.0.0.1:49946)
0925 22:44:58.404251 (+ 1494us) negotiation.cc:107] Waiting for socket to connect
0925 22:44:58.404270 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0925 22:44:58.404368 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:44:58.405366 (+ 998us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:44:58.405369 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:44:58.405387 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:44:58.405481 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:44:58.405487 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:44:58.406860 (+ 1373us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:44:58.406863 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:44:58.409223 (+ 2360us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:44:58.409239 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:44:58.410727 (+ 1488us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:44:58.410735 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:44:58.410849 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:44:58.410896 (+ 47us) client_negotiation.cc:769] Sending connection context
0925 22:44:58.410997 (+ 101us) client_negotiation.cc:240] Negotiation successful
0925 22:44:58.411079 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1403,"mutex_wait_us":899,"thread_start_us":92,"threads_started":1}
I20260925 22:44:58.411294 19062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:44:58.403577 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49946 (local address 127.10.198.2:41061)
0925 22:44:58.404730 (+ 1153us) server_negotiation.cc:206] Beginning negotiation
0925 22:44:58.404743 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0925 22:44:58.404766 (+ 23us) server_negotiation.cc:407] Connection header received
0925 22:44:58.404867 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:44:58.404872 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:44:58.405075 (+ 203us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:44:58.405227 (+ 152us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:44:58.405607 (+ 380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:44:58.406628 (+ 1021us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:44:58.409415 (+ 2787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:44:58.410364 (+ 949us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:44:58.410474 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:44:58.410664 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0925 22:44:58.411046 (+ 382us) server_negotiation.cc:299] Negotiation successful
0925 22:44:58.411151 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1016,"thread_start_us":114,"threads_started":1}
I20260925 22:44:58.412240 19058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:44:58.402410 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.198.1:41409 (local address 127.0.0.1:39244)
0925 22:44:58.403125 (+ 715us) negotiation.cc:107] Waiting for socket to connect
0925 22:44:58.403188 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0925 22:44:58.403331 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:44:58.404798 (+ 1467us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:44:58.404814 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:44:58.404843 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:44:58.404994 (+ 151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:44:58.405000 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:44:58.407250 (+ 2250us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:44:58.407253 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:44:58.408463 (+ 1210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:44:58.408470 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:44:58.409888 (+ 1418us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:44:58.409892 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:44:58.410981 (+ 1089us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:44:58.411943 (+ 962us) client_negotiation.cc:769] Sending connection context
0925 22:44:58.412026 (+ 83us) client_negotiation.cc:240] Negotiation successful
0925 22:44:58.412113 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":513,"mutex_wait_us":963,"thread_start_us":109,"threads_started":1}
I20260925 22:44:58.412400 19060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:44:58.403174 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39244 (local address 127.10.198.1:41409)
0925 22:44:58.404173 (+ 999us) server_negotiation.cc:206] Beginning negotiation
0925 22:44:58.404181 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 22:44:58.404210 (+ 29us) server_negotiation.cc:407] Connection header received
0925 22:44:58.404331 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:44:58.404340 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:44:58.404589 (+ 249us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:44:58.404788 (+ 199us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:44:58.405151 (+ 363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:44:58.407130 (+ 1979us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:44:58.408608 (+ 1478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:44:58.409502 (+ 894us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:44:58.409947 (+ 445us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:44:58.410146 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0925 22:44:58.412143 (+ 1997us) server_negotiation.cc:299] Negotiation successful
0925 22:44:58.412269 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":900,"thread_start_us":95,"threads_started":1}
I20260925 22:44:58.414326 18970 tablet_service.cc:1467] Processing CreateTablet for tablet 95f2e7b2fbab4ac2a2ae151089abf29d (DEFAULT_TABLE table=test-table [id=93f02ee37ae244c985117ac8e0992f1d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 22:44:58.415035 18683 tablet_service.cc:1467] Processing CreateTablet for tablet 95f2e7b2fbab4ac2a2ae151089abf29d (DEFAULT_TABLE table=test-table [id=93f02ee37ae244c985117ac8e0992f1d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 22:44:58.415053 18830 tablet_service.cc:1467] Processing CreateTablet for tablet 95f2e7b2fbab4ac2a2ae151089abf29d (DEFAULT_TABLE table=test-table [id=93f02ee37ae244c985117ac8e0992f1d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 22:44:58.415576 18970 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 95f2e7b2fbab4ac2a2ae151089abf29d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:44:58.416157 18830 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 95f2e7b2fbab4ac2a2ae151089abf29d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:44:58.416287 18683 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 95f2e7b2fbab4ac2a2ae151089abf29d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:44:58.423249 19064 tablet_bootstrap.cc:492] T 95f2e7b2fbab4ac2a2ae151089abf29d P 7d80f46d2eac41e7a9f0b2b2e1972abd: Bootstrap starting.
I20260925 22:44:58.424926 19065 tablet_bootstrap.cc:492] T 95f2e7b2fbab4ac2a2ae151089abf29d P 4e7aa6d387e94dffaff030eb416c66c4: Bootstrap starting.
I20260925 22:44:58.425909 19064 tablet_bootstrap.cc:654] T 95f2e7b2fbab4ac2a2ae151089abf29d P 7d80f46d2eac41e7a9f0b2b2e1972abd: Neither blocks nor log segments found. Creating new log.
I20260925 22:44:58.426789 19064 log.cc:824] T 95f2e7b2fbab4ac2a2ae151089abf29d P 7d80f46d2eac41e7a9f0b2b2e1972abd: Log is configured to *not* fsync() on all Append() calls
I20260925 22:44:58.427296 19065 tablet_bootstrap.cc:654] T 95f2e7b2fbab4ac2a2ae151089abf29d P 4e7aa6d387e94dffaff030eb416c66c4: Neither blocks nor log segments found. Creating new log.
I20260925 22:44:58.428253 19065 log.cc:824] T 95f2e7b2fbab4ac2a2ae151089abf29d P 4e7aa6d387e94dffaff030eb416c66c4: Log is configured to *not* fsync() on all Append() calls
I20260925 22:44:58.428354 19066 tablet_bootstrap.cc:492] T 95f2e7b2fbab4ac2a2ae151089abf29d P fe72d9fd358f4bf7967ac3ea2264aed9: Bootstrap starting.
I20260925 22:44:58.429144 19064 tablet_bootstrap.cc:492] T 95f2e7b2fbab4ac2a2ae151089abf29d P 7d80f46d2eac41e7a9f0b2b2e1972abd: No bootstrap required, opened a new log
I20260925 22:44:58.429375 19064 ts_tablet_manager.cc:1397] T 95f2e7b2fbab4ac2a2ae151089abf29d P 7d80f46d2eac41e7a9f0b2b2e1972abd: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260925 22:44:58.430603 19065 tablet_bootstrap.cc:492] T 95f2e7b2fbab4ac2a2ae151089abf29d P 4e7aa6d387e94dffaff030eb416c66c4: No bootstrap required, opened a new log
I20260925 22:44:58.430809 19066 tablet_bootstrap.cc:654] T 95f2e7b2fbab4ac2a2ae151089abf29d P fe72d9fd358f4bf7967ac3ea2264aed9: Neither blocks nor log segments found. Creating new log.
I20260925 22:44:58.430846 19065 ts_tablet_manager.cc:1397] T 95f2e7b2fbab4ac2a2ae151089abf29d P 4e7aa6d387e94dffaff030eb416c66c4: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260925 22:44:58.431860 19066 log.cc:824] T 95f2e7b2fbab4ac2a2ae151089abf29d P fe72d9fd358f4bf7967ac3ea2264aed9: Log is configured to *not* fsync() on all Append() calls
I20260925 22:44:58.433125 19064 raft_consensus.cc:348] T 95f2e7b2fbab4ac2a2ae151089abf29d P 7d80f46d2eac41e7a9f0b2b2e1972abd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e7aa6d387e94dffaff030eb416c66c4" member_type: VOTER last_known_addr { host: "127.10.198.1" port: 41409 } } peers { permanent_uuid: "fe72d9fd358f4bf7967ac3ea2264aed9" member_type: VOTER last_known_addr { host: "127.10.198.2" port: 41061 } } peers { permanent_uuid: "7d80f46d2eac41e7a9f0b2b2e1972abd" member_type: VOTER last_known_addr { host: "127.10.198.3" port: 37891 } }
I20260925 22:44:58.433450 19064 raft_consensus.cc:374] T 95f2e7b2fbab4ac2a2ae151089abf29d P 7d80f46d2eac41e7a9f0b2b2e1972abd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:44:58.433511 19064 raft_consensus.cc:729] T 95f2e7b2fbab4ac2a2ae151089abf29d P 7d80f46d2eac41e7a9f0b2b2e1972abd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7d80f46d2eac41e7a9f0b2b2e1972abd, State: Initialized, Role: FOLLOWER
I20260925 22:44:58.434070 19064 consensus_queue.cc:260] T 95f2e7b2fbab4ac2a2ae151089abf29d P 7d80f46d2eac41e7a9f0b2b2e1972abd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e7aa6d387e94dffaff030eb416c66c4" member_type: VOTER last_known_addr { host: "127.10.198.1" port: 41409 } } peers { permanent_uuid: "fe72d9fd358f4bf7967ac3ea2264aed9" member_type: VOTER last_known_addr { host: "127.10.198.2" port: 41061 } } peers { permanent_uuid: "7d80f46d2eac41e7a9f0b2b2e1972abd" member_type: VOTER last_known_addr { host: "127.10.198.3" port: 37891 } }
I20260925 22:44:58.434762 19066 tablet_bootstrap.cc:492] T 95f2e7b2fbab4ac2a2ae151089abf29d P fe72d9fd358f4bf7967ac3ea2264aed9: No bootstrap required, opened a new log
I20260925 22:44:58.434813 19036 heartbeater.cc:503] Master 127.10.198.62:46765 was elected leader, sending a full tablet report...
I20260925 22:44:58.435024 19066 ts_tablet_manager.cc:1397] T 95f2e7b2fbab4ac2a2ae151089abf29d P fe72d9fd358f4bf7967ac3ea2264aed9: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260925 22:44:58.435922 19065 raft_consensus.cc:348] T 95f2e7b2fbab4ac2a2ae151089abf29d P 4e7aa6d387e94dffaff030eb416c66c4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e7aa6d387e94dffaff030eb416c66c4" member_type: VOTER last_known_addr { host: "127.10.198.1" port: 41409 } } peers { permanent_uuid: "fe72d9fd358f4bf7967ac3ea2264aed9" member_type: VOTER last_known_addr { host: "127.10.198.2" port: 41061 } } peers { permanent_uuid: "7d80f46d2eac41e7a9f0b2b2e1972abd" member_type: VOTER last_known_addr { host: "127.10.198.3" port: 37891 } }
I20260925 22:44:58.436244 19065 raft_consensus.cc:374] T 95f2e7b2fbab4ac2a2ae151089abf29d P 4e7aa6d387e94dffaff030eb416c66c4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:44:58.436298 19065 raft_consensus.cc:729] T 95f2e7b2fbab4ac2a2ae151089abf29d P 4e7aa6d387e94dffaff030eb416c66c4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4e7aa6d387e94dffaff030eb416c66c4, State: Initialized, Role: FOLLOWER
I20260925 22:44:58.436390 19064 ts_tablet_manager.cc:1428] T 95f2e7b2fbab4ac2a2ae151089abf29d P 7d80f46d2eac41e7a9f0b2b2e1972abd: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260925 22:44:58.436844 19065 consensus_queue.cc:260] T 95f2e7b2fbab4ac2a2ae151089abf29d P 4e7aa6d387e94dffaff030eb416c66c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e7aa6d387e94dffaff030eb416c66c4" member_type: VOTER last_known_addr { host: "127.10.198.1" port: 41409 } } peers { permanent_uuid: "fe72d9fd358f4bf7967ac3ea2264aed9" member_type: VOTER last_known_addr { host: "127.10.198.2" port: 41061 } } peers { permanent_uuid: "7d80f46d2eac41e7a9f0b2b2e1972abd" member_type: VOTER last_known_addr { host: "127.10.198.3" port: 37891 } }
I20260925 22:44:58.437628 18756 heartbeater.cc:503] Master 127.10.198.62:46765 was elected leader, sending a full tablet report...
I20260925 22:44:58.437991 19065 ts_tablet_manager.cc:1428] T 95f2e7b2fbab4ac2a2ae151089abf29d P 4e7aa6d387e94dffaff030eb416c66c4: Time spent starting tablet: real 0.007s user 0.008s sys 0.000s
I20260925 22:44:58.439319 19066 raft_consensus.cc:348] T 95f2e7b2fbab4ac2a2ae151089abf29d P fe72d9fd358f4bf7967ac3ea2264aed9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e7aa6d387e94dffaff030eb416c66c4" member_type: VOTER last_known_addr { host: "127.10.198.1" port: 41409 } } peers { permanent_uuid: "fe72d9fd358f4bf7967ac3ea2264aed9" member_type: VOTER last_known_addr { host: "127.10.198.2" port: 41061 } } peers { permanent_uuid: "7d80f46d2eac41e7a9f0b2b2e1972abd" member_type: VOTER last_known_addr { host: "127.10.198.3" port: 37891 } }
I20260925 22:44:58.439654 19066 raft_consensus.cc:374] T 95f2e7b2fbab4ac2a2ae151089abf29d P fe72d9fd358f4bf7967ac3ea2264aed9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:44:58.439772 19066 raft_consensus.cc:729] T 95f2e7b2fbab4ac2a2ae151089abf29d P fe72d9fd358f4bf7967ac3ea2264aed9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe72d9fd358f4bf7967ac3ea2264aed9, State: Initialized, Role: FOLLOWER
I20260925 22:44:58.440431 19066 consensus_queue.cc:260] T 95f2e7b2fbab4ac2a2ae151089abf29d P fe72d9fd358f4bf7967ac3ea2264aed9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e7aa6d387e94dffaff030eb416c66c4" member_type: VOTER last_known_addr { host: "127.10.198.1" port: 41409 } } peers { permanent_uuid: "fe72d9fd358f4bf7967ac3ea2264aed9" member_type: VOTER last_known_addr { host: "127.10.198.2" port: 41061 } } peers { permanent_uuid: "7d80f46d2eac41e7a9f0b2b2e1972abd" member_type: VOTER last_known_addr { host: "127.10.198.3" port: 37891 } }
I20260925 22:44:58.441329 18896 heartbeater.cc:503] Master 127.10.198.62:46765 was elected leader, sending a full tablet report...
I20260925 22:44:58.441817 19066 ts_tablet_manager.cc:1428] T 95f2e7b2fbab4ac2a2ae151089abf29d P fe72d9fd358f4bf7967ac3ea2264aed9: Time spent starting tablet: real 0.007s user 0.008s sys 0.000s
W20260925 22:44:58.579171 19037 tablet.cc:2369] T 95f2e7b2fbab4ac2a2ae151089abf29d P 7d80f46d2eac41e7a9f0b2b2e1972abd: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 22:44:58.584316 18757 tablet.cc:2369] T 95f2e7b2fbab4ac2a2ae151089abf29d P 4e7aa6d387e94dffaff030eb416c66c4: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 22:44:58.584933 18897 tablet.cc:2369] T 95f2e7b2fbab4ac2a2ae151089abf29d P fe72d9fd358f4bf7967ac3ea2264aed9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 22:44:58.602092 19072 raft_consensus.cc:482] T 95f2e7b2fbab4ac2a2ae151089abf29d P fe72d9fd358f4bf7967ac3ea2264aed9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 22:44:58.602346 19072 raft_consensus.cc:504] T 95f2e7b2fbab4ac2a2ae151089abf29d P fe72d9fd358f4bf7967ac3ea2264aed9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e7aa6d387e94dffaff030eb416c66c4" member_type: VOTER last_known_addr { host: "127.10.198.1" port: 41409 } } peers { permanent_uuid: "fe72d9fd358f4bf7967ac3ea2264aed9" member_type: VOTER last_known_addr { host: "127.10.198.2" port: 41061 } } peers { permanent_uuid: "7d80f46d2eac41e7a9f0b2b2e1972abd" member_type: VOTER last_known_addr { host: "127.10.198.3" port: 37891 } }
I20260925 22:44:58.603657 19072 leader_election.cc:288] T 95f2e7b2fbab4ac2a2ae151089abf29d P fe72d9fd358f4bf7967ac3ea2264aed9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4e7aa6d387e94dffaff030eb416c66c4 (127.10.198.1:41409), 7d80f46d2eac41e7a9f0b2b2e1972abd (127.10.198.3:37891)
I20260925 22:44:58.608212 19063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:44:58.604295 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.198.2:58483 (local address 127.10.198.3:37891)
0925 22:44:58.604453 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0925 22:44:58.604459 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 22:44:58.604657 (+ 198us) server_negotiation.cc:407] Connection header received
0925 22:44:58.604792 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:44:58.604798 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:44:58.604853 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:44:58.604946 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:44:58.605220 (+ 274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:44:58.605747 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:44:58.607033 (+ 1286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:44:58.607491 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:44:58.607648 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:44:58.607716 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0925 22:44:58.607955 (+ 239us) server_negotiation.cc:299] Negotiation successful
0925 22:44:58.608059 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260925 22:44:58.608332 19060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:44:58.604143 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.198.2:60921 (local address 127.10.198.1:41409)
0925 22:44:58.604331 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0925 22:44:58.604336 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:44:58.604441 (+ 105us) server_negotiation.cc:407] Connection header received
0925 22:44:58.604652 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:44:58.604656 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:44:58.604715 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:44:58.604802 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:44:58.605170 (+ 368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:44:58.605712 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:44:58.607030 (+ 1318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:44:58.607567 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:44:58.607705 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:44:58.607789 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0925 22:44:58.608083 (+ 294us) server_negotiation.cc:299] Negotiation successful
0925 22:44:58.608192 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260925 22:44:58.608276 19073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:44:58.603930 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.198.1:41409 (local address 127.10.198.2:60921)
0925 22:44:58.604319 (+ 389us) negotiation.cc:107] Waiting for socket to connect
0925 22:44:58.604340 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0925 22:44:58.604496 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:44:58.604834 (+ 338us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:44:58.604837 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:44:58.604866 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:44:58.605020 (+ 154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:44:58.605032 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:44:58.605862 (+ 830us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:44:58.605867 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:44:58.606874 (+ 1007us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:44:58.606885 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:44:58.607798 (+ 913us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:44:58.607803 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:44:58.607948 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:44:58.607972 (+ 24us) client_negotiation.cc:769] Sending connection context
0925 22:44:58.608053 (+ 81us) client_negotiation.cc:240] Negotiation successful
0925 22:44:58.608130 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":299,"mutex_wait_us":85,"thread_start_us":88,"threads_started":1}
I20260925 22:44:58.608276 19074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:44:58.604166 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.198.3:37891 (local address 127.10.198.2:58483)
0925 22:44:58.604515 (+ 349us) negotiation.cc:107] Waiting for socket to connect
0925 22:44:58.604527 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 22:44:58.604664 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:44:58.604968 (+ 304us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:44:58.604970 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:44:58.604985 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:44:58.605069 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:44:58.605073 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:44:58.605895 (+ 822us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:44:58.605898 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:44:58.606858 (+ 960us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:44:58.606870 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:44:58.607645 (+ 775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:44:58.607649 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:44:58.607775 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:44:58.607803 (+ 28us) client_negotiation.cc:769] Sending connection context
0925 22:44:58.607937 (+ 134us) client_negotiation.cc:240] Negotiation successful
0925 22:44:58.608041 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":274,"mutex_wait_us":52,"thread_start_us":106,"threads_started":1}
I20260925 22:44:58.609113 18705 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "95f2e7b2fbab4ac2a2ae151089abf29d" candidate_uuid: "fe72d9fd358f4bf7967ac3ea2264aed9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4e7aa6d387e94dffaff030eb416c66c4" is_pre_election: true
I20260925 22:44:58.609176 18990 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "95f2e7b2fbab4ac2a2ae151089abf29d" candidate_uuid: "fe72d9fd358f4bf7967ac3ea2264aed9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7d80f46d2eac41e7a9f0b2b2e1972abd" is_pre_election: true
I20260925 22:44:58.609517 18705 raft_consensus.cc:2454] T 95f2e7b2fbab4ac2a2ae151089abf29d P 4e7aa6d387e94dffaff030eb416c66c4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fe72d9fd358f4bf7967ac3ea2264aed9 in term 0.
I20260925 22:44:58.609517 18990 raft_consensus.cc:2454] T 95f2e7b2fbab4ac2a2ae151089abf29d P 7d80f46d2eac41e7a9f0b2b2e1972abd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fe72d9fd358f4bf7967ac3ea2264aed9 in term 0.
I20260925 22:44:58.610119 18783 leader_election.cc:302] T 95f2e7b2fbab4ac2a2ae151089abf29d P fe72d9fd358f4bf7967ac3ea2264aed9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 4e7aa6d387e94dffaff030eb416c66c4, 7d80f46d2eac41e7a9f0b2b2e1972abd, fe72d9fd358f4bf7967ac3ea2264aed9; no voters:
I20260925 22:44:58.610446 19072 raft_consensus.cc:2781] T 95f2e7b2fbab4ac2a2ae151089abf29d P fe72d9fd358f4bf7967ac3ea2264aed9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 22:44:58.610538 19072 raft_consensus.cc:482] T 95f2e7b2fbab4ac2a2ae151089abf29d P fe72d9fd358f4bf7967ac3ea2264aed9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 22:44:58.610600 19072 raft_consensus.cc:3037] T 95f2e7b2fbab4ac2a2ae151089abf29d P fe72d9fd358f4bf7967ac3ea2264aed9 [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:44:58.611795 19072 raft_consensus.cc:504] T 95f2e7b2fbab4ac2a2ae151089abf29d P fe72d9fd358f4bf7967ac3ea2264aed9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e7aa6d387e94dffaff030eb416c66c4" member_type: VOTER last_known_addr { host: "127.10.198.1" port: 41409 } } peers { permanent_uuid: "fe72d9fd358f4bf7967ac3ea2264aed9" member_type: VOTER last_known_addr { host: "127.10.198.2" port: 41061 } } peers { permanent_uuid: "7d80f46d2eac41e7a9f0b2b2e1972abd" member_type: VOTER last_known_addr { host: "127.10.198.3" port: 37891 } }
I20260925 22:44:58.612372 19072 leader_election.cc:288] T 95f2e7b2fbab4ac2a2ae151089abf29d P fe72d9fd358f4bf7967ac3ea2264aed9 [CANDIDATE]: Term 1 election: Requested vote from peers 4e7aa6d387e94dffaff030eb416c66c4 (127.10.198.1:41409), 7d80f46d2eac41e7a9f0b2b2e1972abd (127.10.198.3:37891)
I20260925 22:44:58.612881 18990 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "95f2e7b2fbab4ac2a2ae151089abf29d" candidate_uuid: "fe72d9fd358f4bf7967ac3ea2264aed9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7d80f46d2eac41e7a9f0b2b2e1972abd"
I20260925 22:44:58.612875 18705 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "95f2e7b2fbab4ac2a2ae151089abf29d" candidate_uuid: "fe72d9fd358f4bf7967ac3ea2264aed9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4e7aa6d387e94dffaff030eb416c66c4"
I20260925 22:44:58.613066 18705 raft_consensus.cc:3037] T 95f2e7b2fbab4ac2a2ae151089abf29d P 4e7aa6d387e94dffaff030eb416c66c4 [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:44:58.613025 18990 raft_consensus.cc:3037] T 95f2e7b2fbab4ac2a2ae151089abf29d P 7d80f46d2eac41e7a9f0b2b2e1972abd [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:44:58.614301 18705 raft_consensus.cc:2454] T 95f2e7b2fbab4ac2a2ae151089abf29d P 4e7aa6d387e94dffaff030eb416c66c4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fe72d9fd358f4bf7967ac3ea2264aed9 in term 1.
I20260925 22:44:58.614569 18990 raft_consensus.cc:2454] T 95f2e7b2fbab4ac2a2ae151089abf29d P 7d80f46d2eac41e7a9f0b2b2e1972abd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fe72d9fd358f4bf7967ac3ea2264aed9 in term 1.
I20260925 22:44:58.614830 18783 leader_election.cc:302] T 95f2e7b2fbab4ac2a2ae151089abf29d P fe72d9fd358f4bf7967ac3ea2264aed9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4e7aa6d387e94dffaff030eb416c66c4, fe72d9fd358f4bf7967ac3ea2264aed9; no voters:
I20260925 22:44:58.615152 19072 raft_consensus.cc:2781] T 95f2e7b2fbab4ac2a2ae151089abf29d P fe72d9fd358f4bf7967ac3ea2264aed9 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 22:44:58.615352 19072 raft_consensus.cc:686] T 95f2e7b2fbab4ac2a2ae151089abf29d P fe72d9fd358f4bf7967ac3ea2264aed9 [term 1 LEADER]: Becoming Leader. State: Replica: fe72d9fd358f4bf7967ac3ea2264aed9, State: Running, Role: LEADER
I20260925 22:44:58.615739 19072 consensus_queue.cc:237] T 95f2e7b2fbab4ac2a2ae151089abf29d P fe72d9fd358f4bf7967ac3ea2264aed9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e7aa6d387e94dffaff030eb416c66c4" member_type: VOTER last_known_addr { host: "127.10.198.1" port: 41409 } } peers { permanent_uuid: "fe72d9fd358f4bf7967ac3ea2264aed9" member_type: VOTER last_known_addr { host: "127.10.198.2" port: 41061 } } peers { permanent_uuid: "7d80f46d2eac41e7a9f0b2b2e1972abd" member_type: VOTER last_known_addr { host: "127.10.198.3" port: 37891 } }
I20260925 22:44:58.619495 18553 catalog_manager.cc:5495] T 95f2e7b2fbab4ac2a2ae151089abf29d P fe72d9fd358f4bf7967ac3ea2264aed9 reported cstate change: term changed from 0 to 1, leader changed from <none> to fe72d9fd358f4bf7967ac3ea2264aed9 (127.10.198.2). New cstate: current_term: 1 leader_uuid: "fe72d9fd358f4bf7967ac3ea2264aed9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e7aa6d387e94dffaff030eb416c66c4" member_type: VOTER last_known_addr { host: "127.10.198.1" port: 41409 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fe72d9fd358f4bf7967ac3ea2264aed9" member_type: VOTER last_known_addr { host: "127.10.198.2" port: 41061 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7d80f46d2eac41e7a9f0b2b2e1972abd" member_type: VOTER last_known_addr { host: "127.10.198.3" port: 37891 } health_report { overall_health: UNKNOWN } } }
I20260925 22:44:58.649461 19062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:44:58.645630 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49952 (local address 127.10.198.2:41061)
0925 22:44:58.645785 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0925 22:44:58.645788 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 22:44:58.645805 (+ 17us) server_negotiation.cc:407] Connection header received
0925 22:44:58.645964 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:44:58.645966 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:44:58.646025 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:44:58.646138 (+ 113us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 22:44:58.646359 (+ 221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:44:58.646862 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:44:58.647895 (+ 1033us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:44:58.648167 (+ 272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:44:58.648273 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:44:58.648578 (+ 305us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 22:44:58.649075 (+ 497us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 22:44:58.649170 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0925 22:44:58.649262 (+ 92us) server_negotiation.cc:299] Negotiation successful
0925 22:44:58.649323 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260925 22:44:58.656330 18705 raft_consensus.cc:1261] T 95f2e7b2fbab4ac2a2ae151089abf29d P 4e7aa6d387e94dffaff030eb416c66c4 [term 1 FOLLOWER]: Refusing update from remote peer fe72d9fd358f4bf7967ac3ea2264aed9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 22:44:58.656325 18990 raft_consensus.cc:1261] T 95f2e7b2fbab4ac2a2ae151089abf29d P 7d80f46d2eac41e7a9f0b2b2e1972abd [term 1 FOLLOWER]: Refusing update from remote peer fe72d9fd358f4bf7967ac3ea2264aed9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 22:44:58.657253 19072 consensus_queue.cc:1035] T 95f2e7b2fbab4ac2a2ae151089abf29d P fe72d9fd358f4bf7967ac3ea2264aed9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e7aa6d387e94dffaff030eb416c66c4" member_type: VOTER last_known_addr { host: "127.10.198.1" port: 41409 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 22:44:58.657532 19075 consensus_queue.cc:1035] T 95f2e7b2fbab4ac2a2ae151089abf29d P fe72d9fd358f4bf7967ac3ea2264aed9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7d80f46d2eac41e7a9f0b2b2e1972abd" member_type: VOTER last_known_addr { host: "127.10.198.3" port: 37891 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 22:44:58.667114 19079 mvcc.cc:205] Tried to move back new op lower bound from 7333381319287492608 to 7333381319133564928. Current Snapshot: MvccSnapshot[applied={T|T < 7333381319287492608}]
I20260925 22:44:58.667407 19078 mvcc.cc:205] Tried to move back new op lower bound from 7333381319287492608 to 7333381319133564928. Current Snapshot: MvccSnapshot[applied={T|T < 7333381319287492608 or (T in {7333381319287492608})}]
I20260925 22:44:58.676785 18551 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55532:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 22:44:58.677026 18551 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55532:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 22:44:58.680004 18551 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9a42cb7f7adf47f1b0dd00bf3bfbd491: Sending DeleteTablet for 3 replicas of tablet 95f2e7b2fbab4ac2a2ae151089abf29d
I20260925 22:44:58.680999 18683 tablet_service.cc:1514] Processing DeleteTablet for tablet 95f2e7b2fbab4ac2a2ae151089abf29d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 22:44:58 UTC) from {username='oryx', principal='oryx/127.10.198.62@KRBTEST.COM'} at 127.0.0.1:39244
I20260925 22:44:58.681281 18830 tablet_service.cc:1514] Processing DeleteTablet for tablet 95f2e7b2fbab4ac2a2ae151089abf29d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 22:44:58 UTC) from {username='oryx', principal='oryx/127.10.198.62@KRBTEST.COM'} at 127.0.0.1:49946
I20260925 22:44:58.681617 19088 tablet_replica.cc:330] T 95f2e7b2fbab4ac2a2ae151089abf29d P 4e7aa6d387e94dffaff030eb416c66c4: stopping tablet replica
I20260925 22:44:58.681749 18551 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55532:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260925 22:44:58.681797 18970 tablet_service.cc:1514] Processing DeleteTablet for tablet 95f2e7b2fbab4ac2a2ae151089abf29d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 22:44:58 UTC) from {username='oryx', principal='oryx/127.10.198.62@KRBTEST.COM'} at 127.0.0.1:60530
I20260925 22:44:58.681985 19088 raft_consensus.cc:2229] T 95f2e7b2fbab4ac2a2ae151089abf29d P 4e7aa6d387e94dffaff030eb416c66c4 [term 1 FOLLOWER]: Raft consensus shutting down.
W20260925 22:44:58.682212 18551 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 22:44:58.682334 19088 raft_consensus.cc:2258] T 95f2e7b2fbab4ac2a2ae151089abf29d P 4e7aa6d387e94dffaff030eb416c66c4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 22:44:58.682799 19089 tablet_replica.cc:330] T 95f2e7b2fbab4ac2a2ae151089abf29d P fe72d9fd358f4bf7967ac3ea2264aed9: stopping tablet replica
I20260925 22:44:58.683027 19090 tablet_replica.cc:330] T 95f2e7b2fbab4ac2a2ae151089abf29d P 7d80f46d2eac41e7a9f0b2b2e1972abd: stopping tablet replica
I20260925 22:44:58.683403 19090 raft_consensus.cc:2229] T 95f2e7b2fbab4ac2a2ae151089abf29d P 7d80f46d2eac41e7a9f0b2b2e1972abd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 22:44:58.683809 19090 raft_consensus.cc:2258] T 95f2e7b2fbab4ac2a2ae151089abf29d P 7d80f46d2eac41e7a9f0b2b2e1972abd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 22:44:58.684445 19088 ts_tablet_manager.cc:1905] T 95f2e7b2fbab4ac2a2ae151089abf29d P 4e7aa6d387e94dffaff030eb416c66c4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 22:44:58.685695 19090 ts_tablet_manager.cc:1905] T 95f2e7b2fbab4ac2a2ae151089abf29d P 7d80f46d2eac41e7a9f0b2b2e1972abd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 22:44:58.687567 19088 ts_tablet_manager.cc:1918] T 95f2e7b2fbab4ac2a2ae151089abf29d P 4e7aa6d387e94dffaff030eb416c66c4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 22:44:58.687711 19088 log.cc:1196] T 95f2e7b2fbab4ac2a2ae151089abf29d P 4e7aa6d387e94dffaff030eb416c66c4: Deleting WAL directory at /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790376259650642-11032-0/minicluster-data/ts-0/wal/wals/95f2e7b2fbab4ac2a2ae151089abf29d
I20260925 22:44:58.688139 19089 raft_consensus.cc:2229] T 95f2e7b2fbab4ac2a2ae151089abf29d P fe72d9fd358f4bf7967ac3ea2264aed9 [term 1 LEADER]: Raft consensus shutting down.
I20260925 22:44:58.688691 19089 raft_consensus.cc:2258] T 95f2e7b2fbab4ac2a2ae151089abf29d P fe72d9fd358f4bf7967ac3ea2264aed9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 22:44:58.689285 18683 tablet_service.cc:1467] Processing CreateTablet for tablet 8bea55fc27e746adb1fb1c2e7d4959e8 (DEFAULT_TABLE table=test-table [id=a0575eaaf0aa43158e30e3dcf5bb1564]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 22:44:58.689334 18970 tablet_service.cc:1467] Processing CreateTablet for tablet 8bea55fc27e746adb1fb1c2e7d4959e8 (DEFAULT_TABLE table=test-table [id=a0575eaaf0aa43158e30e3dcf5bb1564]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 22:44:58.689592 18683 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8bea55fc27e746adb1fb1c2e7d4959e8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:44:58.689817 18970 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8bea55fc27e746adb1fb1c2e7d4959e8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:44:58.690361 19089 ts_tablet_manager.cc:1905] T 95f2e7b2fbab4ac2a2ae151089abf29d P fe72d9fd358f4bf7967ac3ea2264aed9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 22:44:58.692180 18830 tablet_service.cc:1467] Processing CreateTablet for tablet 8bea55fc27e746adb1fb1c2e7d4959e8 (DEFAULT_TABLE table=test-table [id=a0575eaaf0aa43158e30e3dcf5bb1564]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 22:44:58.692294 19064 tablet_bootstrap.cc:492] T 8bea55fc27e746adb1fb1c2e7d4959e8 P 7d80f46d2eac41e7a9f0b2b2e1972abd: Bootstrap starting.
I20260925 22:44:58.692513 18830 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8bea55fc27e746adb1fb1c2e7d4959e8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:44:58.693112 19090 ts_tablet_manager.cc:1918] T 95f2e7b2fbab4ac2a2ae151089abf29d P 7d80f46d2eac41e7a9f0b2b2e1972abd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 22:44:58.693305 19090 log.cc:1196] T 95f2e7b2fbab4ac2a2ae151089abf29d P 7d80f46d2eac41e7a9f0b2b2e1972abd: Deleting WAL directory at /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790376259650642-11032-0/minicluster-data/ts-2/wal/wals/95f2e7b2fbab4ac2a2ae151089abf29d
I20260925 22:44:58.693365 19089 ts_tablet_manager.cc:1918] T 95f2e7b2fbab4ac2a2ae151089abf29d P fe72d9fd358f4bf7967ac3ea2264aed9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 22:44:58.693466 19064 tablet_bootstrap.cc:654] T 8bea55fc27e746adb1fb1c2e7d4959e8 P 7d80f46d2eac41e7a9f0b2b2e1972abd: Neither blocks nor log segments found. Creating new log.
I20260925 22:44:58.693485 19089 log.cc:1196] T 95f2e7b2fbab4ac2a2ae151089abf29d P fe72d9fd358f4bf7967ac3ea2264aed9: Deleting WAL directory at /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790376259650642-11032-0/minicluster-data/ts-1/wal/wals/95f2e7b2fbab4ac2a2ae151089abf29d
I20260925 22:44:58.693859 19090 ts_tablet_manager.cc:1939] T 95f2e7b2fbab4ac2a2ae151089abf29d P 7d80f46d2eac41e7a9f0b2b2e1972abd: Deleting consensus metadata
I20260925 22:44:58.693859 19089 ts_tablet_manager.cc:1939] T 95f2e7b2fbab4ac2a2ae151089abf29d P fe72d9fd358f4bf7967ac3ea2264aed9: Deleting consensus metadata
I20260925 22:44:58.695010 19066 tablet_bootstrap.cc:492] T 8bea55fc27e746adb1fb1c2e7d4959e8 P fe72d9fd358f4bf7967ac3ea2264aed9: Bootstrap starting.
I20260925 22:44:58.695276 19088 ts_tablet_manager.cc:1939] T 95f2e7b2fbab4ac2a2ae151089abf29d P 4e7aa6d387e94dffaff030eb416c66c4: Deleting consensus metadata
I20260925 22:44:58.695312 18540 catalog_manager.cc:4841] TS 7d80f46d2eac41e7a9f0b2b2e1972abd (127.10.198.3:37891): tablet 95f2e7b2fbab4ac2a2ae151089abf29d (table test-table [id=93f02ee37ae244c985117ac8e0992f1d]) successfully deleted
I20260925 22:44:58.695628 18540 catalog_manager.cc:4841] TS fe72d9fd358f4bf7967ac3ea2264aed9 (127.10.198.2:41061): tablet 95f2e7b2fbab4ac2a2ae151089abf29d (table test-table [id=93f02ee37ae244c985117ac8e0992f1d]) successfully deleted
I20260925 22:44:58.695930 19065 tablet_bootstrap.cc:492] T 8bea55fc27e746adb1fb1c2e7d4959e8 P 4e7aa6d387e94dffaff030eb416c66c4: Bootstrap starting.
I20260925 22:44:58.696242 19066 tablet_bootstrap.cc:654] T 8bea55fc27e746adb1fb1c2e7d4959e8 P fe72d9fd358f4bf7967ac3ea2264aed9: Neither blocks nor log segments found. Creating new log.
I20260925 22:44:58.697074 19065 tablet_bootstrap.cc:654] T 8bea55fc27e746adb1fb1c2e7d4959e8 P 4e7aa6d387e94dffaff030eb416c66c4: Neither blocks nor log segments found. Creating new log.
I20260925 22:44:58.697868 19066 tablet_bootstrap.cc:492] T 8bea55fc27e746adb1fb1c2e7d4959e8 P fe72d9fd358f4bf7967ac3ea2264aed9: No bootstrap required, opened a new log
I20260925 22:44:58.697980 19066 ts_tablet_manager.cc:1397] T 8bea55fc27e746adb1fb1c2e7d4959e8 P fe72d9fd358f4bf7967ac3ea2264aed9: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260925 22:44:58.698493 19066 raft_consensus.cc:348] T 8bea55fc27e746adb1fb1c2e7d4959e8 P fe72d9fd358f4bf7967ac3ea2264aed9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e7aa6d387e94dffaff030eb416c66c4" member_type: VOTER last_known_addr { host: "127.10.198.1" port: 41409 } } peers { permanent_uuid: "fe72d9fd358f4bf7967ac3ea2264aed9" member_type: VOTER last_known_addr { host: "127.10.198.2" port: 41061 } } peers { permanent_uuid: "7d80f46d2eac41e7a9f0b2b2e1972abd" member_type: VOTER last_known_addr { host: "127.10.198.3" port: 37891 } }
I20260925 22:44:58.698613 19066 raft_consensus.cc:374] T 8bea55fc27e746adb1fb1c2e7d4959e8 P fe72d9fd358f4bf7967ac3ea2264aed9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:44:58.698645 19066 raft_consensus.cc:729] T 8bea55fc27e746adb1fb1c2e7d4959e8 P fe72d9fd358f4bf7967ac3ea2264aed9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe72d9fd358f4bf7967ac3ea2264aed9, State: Initialized, Role: FOLLOWER
I20260925 22:44:58.699151 19066 consensus_queue.cc:260] T 8bea55fc27e746adb1fb1c2e7d4959e8 P fe72d9fd358f4bf7967ac3ea2264aed9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e7aa6d387e94dffaff030eb416c66c4" member_type: VOTER last_known_addr { host: "127.10.198.1" port: 41409 } } peers { permanent_uuid: "fe72d9fd358f4bf7967ac3ea2264aed9" member_type: VOTER last_known_addr { host: "127.10.198.2" port: 41061 } } peers { permanent_uuid: "7d80f46d2eac41e7a9f0b2b2e1972abd" member_type: VOTER last_known_addr { host: "127.10.198.3" port: 37891 } }
I20260925 22:44:58.699470 19065 tablet_bootstrap.cc:492] T 8bea55fc27e746adb1fb1c2e7d4959e8 P 4e7aa6d387e94dffaff030eb416c66c4: No bootstrap required, opened a new log
I20260925 22:44:58.699566 19065 ts_tablet_manager.cc:1397] T 8bea55fc27e746adb1fb1c2e7d4959e8 P 4e7aa6d387e94dffaff030eb416c66c4: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260925 22:44:58.699975 18539 catalog_manager.cc:4841] TS 4e7aa6d387e94dffaff030eb416c66c4 (127.10.198.1:41409): tablet 95f2e7b2fbab4ac2a2ae151089abf29d (table test-table [id=93f02ee37ae244c985117ac8e0992f1d]) successfully deleted
I20260925 22:44:58.700114 19065 raft_consensus.cc:348] T 8bea55fc27e746adb1fb1c2e7d4959e8 P 4e7aa6d387e94dffaff030eb416c66c4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e7aa6d387e94dffaff030eb416c66c4" member_type: VOTER last_known_addr { host: "127.10.198.1" port: 41409 } } peers { permanent_uuid: "fe72d9fd358f4bf7967ac3ea2264aed9" member_type: VOTER last_known_addr { host: "127.10.198.2" port: 41061 } } peers { permanent_uuid: "7d80f46d2eac41e7a9f0b2b2e1972abd" member_type: VOTER last_known_addr { host: "127.10.198.3" port: 37891 } }
I20260925 22:44:58.700255 19065 raft_consensus.cc:374] T 8bea55fc27e746adb1fb1c2e7d4959e8 P 4e7aa6d387e94dffaff030eb416c66c4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:44:58.700310 19065 raft_consensus.cc:729] T 8bea55fc27e746adb1fb1c2e7d4959e8 P 4e7aa6d387e94dffaff030eb416c66c4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4e7aa6d387e94dffaff030eb416c66c4, State: Initialized, Role: FOLLOWER
I20260925 22:44:58.700439 19065 consensus_queue.cc:260] T 8bea55fc27e746adb1fb1c2e7d4959e8 P 4e7aa6d387e94dffaff030eb416c66c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e7aa6d387e94dffaff030eb416c66c4" member_type: VOTER last_known_addr { host: "127.10.198.1" port: 41409 } } peers { permanent_uuid: "fe72d9fd358f4bf7967ac3ea2264aed9" member_type: VOTER last_known_addr { host: "127.10.198.2" port: 41061 } } peers { permanent_uuid: "7d80f46d2eac41e7a9f0b2b2e1972abd" member_type: VOTER last_known_addr { host: "127.10.198.3" port: 37891 } }
I20260925 22:44:58.700628 19066 ts_tablet_manager.cc:1428] T 8bea55fc27e746adb1fb1c2e7d4959e8 P fe72d9fd358f4bf7967ac3ea2264aed9: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20260925 22:44:58.700717 19065 ts_tablet_manager.cc:1428] T 8bea55fc27e746adb1fb1c2e7d4959e8 P 4e7aa6d387e94dffaff030eb416c66c4: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260925 22:44:58.702454 19064 tablet_bootstrap.cc:492] T 8bea55fc27e746adb1fb1c2e7d4959e8 P 7d80f46d2eac41e7a9f0b2b2e1972abd: No bootstrap required, opened a new log
I20260925 22:44:58.702574 19064 ts_tablet_manager.cc:1397] T 8bea55fc27e746adb1fb1c2e7d4959e8 P 7d80f46d2eac41e7a9f0b2b2e1972abd: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.000s
I20260925 22:44:58.703218 19064 raft_consensus.cc:348] T 8bea55fc27e746adb1fb1c2e7d4959e8 P 7d80f46d2eac41e7a9f0b2b2e1972abd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e7aa6d387e94dffaff030eb416c66c4" member_type: VOTER last_known_addr { host: "127.10.198.1" port: 41409 } } peers { permanent_uuid: "fe72d9fd358f4bf7967ac3ea2264aed9" member_type: VOTER last_known_addr { host: "127.10.198.2" port: 41061 } } peers { permanent_uuid: "7d80f46d2eac41e7a9f0b2b2e1972abd" member_type: VOTER last_known_addr { host: "127.10.198.3" port: 37891 } }
I20260925 22:44:58.703379 19064 raft_consensus.cc:374] T 8bea55fc27e746adb1fb1c2e7d4959e8 P 7d80f46d2eac41e7a9f0b2b2e1972abd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:44:58.703413 19064 raft_consensus.cc:729] T 8bea55fc27e746adb1fb1c2e7d4959e8 P 7d80f46d2eac41e7a9f0b2b2e1972abd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7d80f46d2eac41e7a9f0b2b2e1972abd, State: Initialized, Role: FOLLOWER
I20260925 22:44:58.703555 19064 consensus_queue.cc:260] T 8bea55fc27e746adb1fb1c2e7d4959e8 P 7d80f46d2eac41e7a9f0b2b2e1972abd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e7aa6d387e94dffaff030eb416c66c4" member_type: VOTER last_known_addr { host: "127.10.198.1" port: 41409 } } peers { permanent_uuid: "fe72d9fd358f4bf7967ac3ea2264aed9" member_type: VOTER last_known_addr { host: "127.10.198.2" port: 41061 } } peers { permanent_uuid: "7d80f46d2eac41e7a9f0b2b2e1972abd" member_type: VOTER last_known_addr { host: "127.10.198.3" port: 37891 } }
I20260925 22:44:58.703902 19064 ts_tablet_manager.cc:1428] T 8bea55fc27e746adb1fb1c2e7d4959e8 P 7d80f46d2eac41e7a9f0b2b2e1972abd: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260925 22:44:58.835958 19071 raft_consensus.cc:482] T 8bea55fc27e746adb1fb1c2e7d4959e8 P 4e7aa6d387e94dffaff030eb416c66c4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 22:44:58.836130 19071 raft_consensus.cc:504] T 8bea55fc27e746adb1fb1c2e7d4959e8 P 4e7aa6d387e94dffaff030eb416c66c4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e7aa6d387e94dffaff030eb416c66c4" member_type: VOTER last_known_addr { host: "127.10.198.1" port: 41409 } } peers { permanent_uuid: "fe72d9fd358f4bf7967ac3ea2264aed9" member_type: VOTER last_known_addr { host: "127.10.198.2" port: 41061 } } peers { permanent_uuid: "7d80f46d2eac41e7a9f0b2b2e1972abd" member_type: VOTER last_known_addr { host: "127.10.198.3" port: 37891 } }
I20260925 22:44:58.837493 19071 leader_election.cc:288] T 8bea55fc27e746adb1fb1c2e7d4959e8 P 4e7aa6d387e94dffaff030eb416c66c4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fe72d9fd358f4bf7967ac3ea2264aed9 (127.10.198.2:41061), 7d80f46d2eac41e7a9f0b2b2e1972abd (127.10.198.3:37891)
I20260925 22:44:58.843163 19062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:44:58.837918 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.198.1:53779 (local address 127.10.198.2:41061)
0925 22:44:58.838060 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0925 22:44:58.838065 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:44:58.838692 (+ 627us) server_negotiation.cc:407] Connection header received
0925 22:44:58.838840 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:44:58.838844 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:44:58.838921 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:44:58.839018 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:44:58.839402 (+ 384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:44:58.840159 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:44:58.841710 (+ 1551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:44:58.842347 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:44:58.842462 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:44:58.842518 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0925 22:44:58.842944 (+ 426us) server_negotiation.cc:299] Negotiation successful
0925 22:44:58.843023 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260925 22:44:58.843191 19094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:44:58.837802 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.198.2:41061 (local address 127.10.198.1:53779)
0925 22:44:58.838577 (+ 775us) negotiation.cc:107] Waiting for socket to connect
0925 22:44:58.838593 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 22:44:58.838741 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:44:58.839149 (+ 408us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:44:58.839152 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:44:58.839172 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:44:58.839267 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:44:58.839273 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:44:58.840341 (+ 1068us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:44:58.840345 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:44:58.841544 (+ 1199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:44:58.841557 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:44:58.842692 (+ 1135us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:44:58.842695 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:44:58.842784 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:44:58.842803 (+ 19us) client_negotiation.cc:769] Sending connection context
0925 22:44:58.842923 (+ 120us) client_negotiation.cc:240] Negotiation successful
0925 22:44:58.843009 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":700,"thread_start_us":88,"threads_started":1}
I20260925 22:44:58.843971 19063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:44:58.838302 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.198.1:58425 (local address 127.10.198.3:37891)
0925 22:44:58.838463 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0925 22:44:58.838468 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:44:58.839066 (+ 598us) server_negotiation.cc:407] Connection header received
0925 22:44:58.839270 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:44:58.839273 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:44:58.839324 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:44:58.839377 (+ 53us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:44:58.839823 (+ 446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:44:58.840644 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:44:58.842230 (+ 1586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:44:58.842855 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:44:58.842968 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:44:58.843016 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0925 22:44:58.843681 (+ 665us) server_negotiation.cc:299] Negotiation successful
0925 22:44:58.843779 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260925 22:44:58.843920 18850 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8bea55fc27e746adb1fb1c2e7d4959e8" candidate_uuid: "4e7aa6d387e94dffaff030eb416c66c4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe72d9fd358f4bf7967ac3ea2264aed9" is_pre_election: true
I20260925 22:44:58.844214 18850 raft_consensus.cc:2454] T 8bea55fc27e746adb1fb1c2e7d4959e8 P fe72d9fd358f4bf7967ac3ea2264aed9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4e7aa6d387e94dffaff030eb416c66c4 in term 0.
I20260925 22:44:58.844722 19095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:44:58.838169 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.198.3:37891 (local address 127.10.198.1:58425)
0925 22:44:58.838989 (+ 820us) negotiation.cc:107] Waiting for socket to connect
0925 22:44:58.839002 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 22:44:58.839144 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:44:58.839549 (+ 405us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:44:58.839552 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:44:58.839568 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:44:58.839671 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:44:58.839677 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:44:58.840810 (+ 1133us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:44:58.840814 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:44:58.842076 (+ 1262us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:44:58.842095 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:44:58.843186 (+ 1091us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:44:58.843190 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:44:58.843508 (+ 318us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:44:58.843543 (+ 35us) client_negotiation.cc:769] Sending connection context
0925 22:44:58.843630 (+ 87us) client_negotiation.cc:240] Negotiation successful
0925 22:44:58.844588 (+ 958us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":761,"thread_start_us":89,"threads_started":1}
I20260925 22:44:58.844713 18644 leader_election.cc:302] T 8bea55fc27e746adb1fb1c2e7d4959e8 P 4e7aa6d387e94dffaff030eb416c66c4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4e7aa6d387e94dffaff030eb416c66c4, fe72d9fd358f4bf7967ac3ea2264aed9; no voters:
I20260925 22:44:58.845125 19071 raft_consensus.cc:2781] T 8bea55fc27e746adb1fb1c2e7d4959e8 P 4e7aa6d387e94dffaff030eb416c66c4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 22:44:58.845187 19071 raft_consensus.cc:482] T 8bea55fc27e746adb1fb1c2e7d4959e8 P 4e7aa6d387e94dffaff030eb416c66c4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 22:44:58.845223 19071 raft_consensus.cc:3037] T 8bea55fc27e746adb1fb1c2e7d4959e8 P 4e7aa6d387e94dffaff030eb416c66c4 [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:44:58.845342 18990 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8bea55fc27e746adb1fb1c2e7d4959e8" candidate_uuid: "4e7aa6d387e94dffaff030eb416c66c4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7d80f46d2eac41e7a9f0b2b2e1972abd" is_pre_election: true
I20260925 22:44:58.845525 18990 raft_consensus.cc:2454] T 8bea55fc27e746adb1fb1c2e7d4959e8 P 7d80f46d2eac41e7a9f0b2b2e1972abd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4e7aa6d387e94dffaff030eb416c66c4 in term 0.
I20260925 22:44:58.846513 19071 raft_consensus.cc:504] T 8bea55fc27e746adb1fb1c2e7d4959e8 P 4e7aa6d387e94dffaff030eb416c66c4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e7aa6d387e94dffaff030eb416c66c4" member_type: VOTER last_known_addr { host: "127.10.198.1" port: 41409 } } peers { permanent_uuid: "fe72d9fd358f4bf7967ac3ea2264aed9" member_type: VOTER last_known_addr { host: "127.10.198.2" port: 41061 } } peers { permanent_uuid: "7d80f46d2eac41e7a9f0b2b2e1972abd" member_type: VOTER last_known_addr { host: "127.10.198.3" port: 37891 } }
I20260925 22:44:58.846913 19071 leader_election.cc:288] T 8bea55fc27e746adb1fb1c2e7d4959e8 P 4e7aa6d387e94dffaff030eb416c66c4 [CANDIDATE]: Term 1 election: Requested vote from peers fe72d9fd358f4bf7967ac3ea2264aed9 (127.10.198.2:41061), 7d80f46d2eac41e7a9f0b2b2e1972abd (127.10.198.3:37891)
I20260925 22:44:58.847417 18850 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8bea55fc27e746adb1fb1c2e7d4959e8" candidate_uuid: "4e7aa6d387e94dffaff030eb416c66c4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe72d9fd358f4bf7967ac3ea2264aed9"
I20260925 22:44:58.847447 18990 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8bea55fc27e746adb1fb1c2e7d4959e8" candidate_uuid: "4e7aa6d387e94dffaff030eb416c66c4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7d80f46d2eac41e7a9f0b2b2e1972abd"
I20260925 22:44:58.847572 18850 raft_consensus.cc:3037] T 8bea55fc27e746adb1fb1c2e7d4959e8 P fe72d9fd358f4bf7967ac3ea2264aed9 [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:44:58.847611 18990 raft_consensus.cc:3037] T 8bea55fc27e746adb1fb1c2e7d4959e8 P 7d80f46d2eac41e7a9f0b2b2e1972abd [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:44:58.848678 18990 raft_consensus.cc:2454] T 8bea55fc27e746adb1fb1c2e7d4959e8 P 7d80f46d2eac41e7a9f0b2b2e1972abd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4e7aa6d387e94dffaff030eb416c66c4 in term 1.
I20260925 22:44:58.848703 18850 raft_consensus.cc:2454] T 8bea55fc27e746adb1fb1c2e7d4959e8 P fe72d9fd358f4bf7967ac3ea2264aed9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4e7aa6d387e94dffaff030eb416c66c4 in term 1.
I20260925 22:44:58.849053 18644 leader_election.cc:302] T 8bea55fc27e746adb1fb1c2e7d4959e8 P 4e7aa6d387e94dffaff030eb416c66c4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4e7aa6d387e94dffaff030eb416c66c4, fe72d9fd358f4bf7967ac3ea2264aed9; no voters:
I20260925 22:44:58.849316 19071 raft_consensus.cc:2781] T 8bea55fc27e746adb1fb1c2e7d4959e8 P 4e7aa6d387e94dffaff030eb416c66c4 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 22:44:58.849572 19071 raft_consensus.cc:686] T 8bea55fc27e746adb1fb1c2e7d4959e8 P 4e7aa6d387e94dffaff030eb416c66c4 [term 1 LEADER]: Becoming Leader. State: Replica: 4e7aa6d387e94dffaff030eb416c66c4, State: Running, Role: LEADER
I20260925 22:44:58.850098 19071 consensus_queue.cc:237] T 8bea55fc27e746adb1fb1c2e7d4959e8 P 4e7aa6d387e94dffaff030eb416c66c4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e7aa6d387e94dffaff030eb416c66c4" member_type: VOTER last_known_addr { host: "127.10.198.1" port: 41409 } } peers { permanent_uuid: "fe72d9fd358f4bf7967ac3ea2264aed9" member_type: VOTER last_known_addr { host: "127.10.198.2" port: 41061 } } peers { permanent_uuid: "7d80f46d2eac41e7a9f0b2b2e1972abd" member_type: VOTER last_known_addr { host: "127.10.198.3" port: 37891 } }
I20260925 22:44:58.852722 18551 catalog_manager.cc:5495] T 8bea55fc27e746adb1fb1c2e7d4959e8 P 4e7aa6d387e94dffaff030eb416c66c4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4e7aa6d387e94dffaff030eb416c66c4 (127.10.198.1). New cstate: current_term: 1 leader_uuid: "4e7aa6d387e94dffaff030eb416c66c4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e7aa6d387e94dffaff030eb416c66c4" member_type: VOTER last_known_addr { host: "127.10.198.1" port: 41409 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fe72d9fd358f4bf7967ac3ea2264aed9" member_type: VOTER last_known_addr { host: "127.10.198.2" port: 41061 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7d80f46d2eac41e7a9f0b2b2e1972abd" member_type: VOTER last_known_addr { host: "127.10.198.3" port: 37891 } health_report { overall_health: UNKNOWN } } }
Sep 25 22:44:58 dist-test-slave-5vdr krb5kdc[18508](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376297, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.198.62@KRBTEST.COM for oryx/127.10.198.62@KRBTEST.COM
I20260925 22:44:58.899762 19103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:44:58.885813 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.198.62:46765 (local address 127.0.0.1:55548)
0925 22:44:58.886260 (+ 447us) negotiation.cc:107] Waiting for socket to connect
0925 22:44:58.886278 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0925 22:44:58.886424 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:44:58.886704 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:44:58.886706 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:44:58.887163 (+ 457us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:44:58.887263 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:44:58.887269 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:44:58.888160 (+ 891us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:44:58.888163 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:44:58.888728 (+ 565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:44:58.888735 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:44:58.889245 (+ 510us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:44:58.889248 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:44:58.889332 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:44:58.890048 (+ 716us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:44:58.890062 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:44:58.892218 (+ 2156us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:44:58.896447 (+ 4229us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:44:58.896469 (+ 22us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:44:58.896479 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:44:58.897106 (+ 627us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:44:58.897938 (+ 832us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:44:58.897943 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:44:58.897948 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:44:58.898061 (+ 113us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:44:58.898881 (+ 820us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:44:58.898893 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:44:58.899120 (+ 227us) client_negotiation.cc:769] Sending connection context
0925 22:44:58.899273 (+ 153us) client_negotiation.cc:240] Negotiation successful
0925 22:44:58.899554 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":317,"thread_start_us":157,"threads_started":1}
I20260925 22:44:58.899762 19104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:44:58.885968 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55548 (local address 127.10.198.62:46765)
0925 22:44:58.886260 (+ 292us) server_negotiation.cc:206] Beginning negotiation
0925 22:44:58.886266 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 22:44:58.886366 (+ 100us) server_negotiation.cc:407] Connection header received
0925 22:44:58.886529 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:44:58.886533 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:44:58.886581 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:44:58.886686 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:44:58.887399 (+ 713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:44:58.888017 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:44:58.888856 (+ 839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:44:58.889083 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:44:58.889211 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:44:58.892456 (+ 3245us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:44:58.892461 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:44:58.892466 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:44:58.892501 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:44:58.896259 (+ 3758us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:44:58.897216 (+ 957us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:44:58.897220 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:44:58.897222 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:44:58.897793 (+ 571us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:44:58.898311 (+ 518us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:44:58.898317 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:44:58.898321 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:44:58.898668 (+ 347us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:44:58.898864 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0925 22:44:58.899332 (+ 468us) server_negotiation.cc:299] Negotiation successful
0925 22:44:58.899576 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":123,"server-negotiator.queue_time_us":242,"thread_start_us":58,"threads_started":1}
I20260925 22:44:58.905079 18551 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.10.198.62@KRBTEST.COM'} at 127.0.0.1:55548:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260925 22:44:58.905895 18551 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 22:44:58.912350 18683 tablet_service.cc:1467] Processing CreateTablet for tablet e4c98185d11946379fbd371d55c76871 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=655d902bf76240a2a4e4fd0208b524f9]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260925 22:44:58.912832 18683 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e4c98185d11946379fbd371d55c76871. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:44:58.912732 18830 tablet_service.cc:1467] Processing CreateTablet for tablet e4c98185d11946379fbd371d55c76871 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=655d902bf76240a2a4e4fd0208b524f9]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260925 22:44:58.913158 18830 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e4c98185d11946379fbd371d55c76871. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:44:58.915710 19065 tablet_bootstrap.cc:492] T e4c98185d11946379fbd371d55c76871 P 4e7aa6d387e94dffaff030eb416c66c4: Bootstrap starting.
I20260925 22:44:58.916730 19065 tablet_bootstrap.cc:654] T e4c98185d11946379fbd371d55c76871 P 4e7aa6d387e94dffaff030eb416c66c4: Neither blocks nor log segments found. Creating new log.
I20260925 22:44:58.917030 19066 tablet_bootstrap.cc:492] T e4c98185d11946379fbd371d55c76871 P fe72d9fd358f4bf7967ac3ea2264aed9: Bootstrap starting.
I20260925 22:44:58.917577 18970 tablet_service.cc:1467] Processing CreateTablet for tablet e4c98185d11946379fbd371d55c76871 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=655d902bf76240a2a4e4fd0208b524f9]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260925 22:44:58.917945 19066 tablet_bootstrap.cc:654] T e4c98185d11946379fbd371d55c76871 P fe72d9fd358f4bf7967ac3ea2264aed9: Neither blocks nor log segments found. Creating new log.
I20260925 22:44:58.918027 18970 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e4c98185d11946379fbd371d55c76871. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:44:58.919086 19066 tablet_bootstrap.cc:492] T e4c98185d11946379fbd371d55c76871 P fe72d9fd358f4bf7967ac3ea2264aed9: No bootstrap required, opened a new log
I20260925 22:44:58.919157 19066 ts_tablet_manager.cc:1397] T e4c98185d11946379fbd371d55c76871 P fe72d9fd358f4bf7967ac3ea2264aed9: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260925 22:44:58.919486 19066 raft_consensus.cc:348] T e4c98185d11946379fbd371d55c76871 P fe72d9fd358f4bf7967ac3ea2264aed9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e7aa6d387e94dffaff030eb416c66c4" member_type: VOTER last_known_addr { host: "127.10.198.1" port: 41409 } } peers { permanent_uuid: "fe72d9fd358f4bf7967ac3ea2264aed9" member_type: VOTER last_known_addr { host: "127.10.198.2" port: 41061 } } peers { permanent_uuid: "7d80f46d2eac41e7a9f0b2b2e1972abd" member_type: VOTER last_known_addr { host: "127.10.198.3" port: 37891 } }
I20260925 22:44:58.919576 19066 raft_consensus.cc:374] T e4c98185d11946379fbd371d55c76871 P fe72d9fd358f4bf7967ac3ea2264aed9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:44:58.919597 19066 raft_consensus.cc:729] T e4c98185d11946379fbd371d55c76871 P fe72d9fd358f4bf7967ac3ea2264aed9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe72d9fd358f4bf7967ac3ea2264aed9, State: Initialized, Role: FOLLOWER
I20260925 22:44:58.919744 19066 consensus_queue.cc:260] T e4c98185d11946379fbd371d55c76871 P fe72d9fd358f4bf7967ac3ea2264aed9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e7aa6d387e94dffaff030eb416c66c4" member_type: VOTER last_known_addr { host: "127.10.198.1" port: 41409 } } peers { permanent_uuid: "fe72d9fd358f4bf7967ac3ea2264aed9" member_type: VOTER last_known_addr { host: "127.10.198.2" port: 41061 } } peers { permanent_uuid: "7d80f46d2eac41e7a9f0b2b2e1972abd" member_type: VOTER last_known_addr { host: "127.10.198.3" port: 37891 } }
I20260925 22:44:58.920081 19066 ts_tablet_manager.cc:1428] T e4c98185d11946379fbd371d55c76871 P fe72d9fd358f4bf7967ac3ea2264aed9: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260925 22:44:58.920024 19083 tablet_replica.cc:439] T e4c98185d11946379fbd371d55c76871 P fe72d9fd358f4bf7967ac3ea2264aed9: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e7aa6d387e94dffaff030eb416c66c4" member_type: VOTER last_known_addr { host: "127.10.198.1" port: 41409 } } peers { permanent_uuid: "fe72d9fd358f4bf7967ac3ea2264aed9" member_type: VOTER last_known_addr { host: "127.10.198.2" port: 41061 } } peers { permanent_uuid: "7d80f46d2eac41e7a9f0b2b2e1972abd" member_type: VOTER last_known_addr { host: "127.10.198.3" port: 37891 } } }
I20260925 22:44:58.920161 19083 tablet_replica.cc:442] T e4c98185d11946379fbd371d55c76871 P fe72d9fd358f4bf7967ac3ea2264aed9: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 22:44:58.923435 19064 tablet_bootstrap.cc:492] T e4c98185d11946379fbd371d55c76871 P 7d80f46d2eac41e7a9f0b2b2e1972abd: Bootstrap starting.
I20260925 22:44:58.924543 19065 tablet_bootstrap.cc:492] T e4c98185d11946379fbd371d55c76871 P 4e7aa6d387e94dffaff030eb416c66c4: No bootstrap required, opened a new log
I20260925 22:44:58.924650 19064 tablet_bootstrap.cc:654] T e4c98185d11946379fbd371d55c76871 P 7d80f46d2eac41e7a9f0b2b2e1972abd: Neither blocks nor log segments found. Creating new log.
I20260925 22:44:58.924675 19065 ts_tablet_manager.cc:1397] T e4c98185d11946379fbd371d55c76871 P 4e7aa6d387e94dffaff030eb416c66c4: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.000s
I20260925 22:44:58.925217 19065 raft_consensus.cc:348] T e4c98185d11946379fbd371d55c76871 P 4e7aa6d387e94dffaff030eb416c66c4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e7aa6d387e94dffaff030eb416c66c4" member_type: VOTER last_known_addr { host: "127.10.198.1" port: 41409 } } peers { permanent_uuid: "fe72d9fd358f4bf7967ac3ea2264aed9" member_type: VOTER last_known_addr { host: "127.10.198.2" port: 41061 } } peers { permanent_uuid: "7d80f46d2eac41e7a9f0b2b2e1972abd" member_type: VOTER last_known_addr { host: "127.10.198.3" port: 37891 } }
I20260925 22:44:58.925387 19065 raft_consensus.cc:374] T e4c98185d11946379fbd371d55c76871 P 4e7aa6d387e94dffaff030eb416c66c4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:44:58.925426 19065 raft_consensus.cc:729] T e4c98185d11946379fbd371d55c76871 P 4e7aa6d387e94dffaff030eb416c66c4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4e7aa6d387e94dffaff030eb416c66c4, State: Initialized, Role: FOLLOWER
I20260925 22:44:58.925595 19065 consensus_queue.cc:260] T e4c98185d11946379fbd371d55c76871 P 4e7aa6d387e94dffaff030eb416c66c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e7aa6d387e94dffaff030eb416c66c4" member_type: VOTER last_known_addr { host: "127.10.198.1" port: 41409 } } peers { permanent_uuid: "fe72d9fd358f4bf7967ac3ea2264aed9" member_type: VOTER last_known_addr { host: "127.10.198.2" port: 41061 } } peers { permanent_uuid: "7d80f46d2eac41e7a9f0b2b2e1972abd" member_type: VOTER last_known_addr { host: "127.10.198.3" port: 37891 } }
I20260925 22:44:58.925956 19065 ts_tablet_manager.cc:1428] T e4c98185d11946379fbd371d55c76871 P 4e7aa6d387e94dffaff030eb416c66c4: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20260925 22:44:58.926098 19064 tablet_bootstrap.cc:492] T e4c98185d11946379fbd371d55c76871 P 7d80f46d2eac41e7a9f0b2b2e1972abd: No bootstrap required, opened a new log
I20260925 22:44:58.926198 19064 ts_tablet_manager.cc:1397] T e4c98185d11946379fbd371d55c76871 P 7d80f46d2eac41e7a9f0b2b2e1972abd: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260925 22:44:58.926219 19071 tablet_replica.cc:439] T e4c98185d11946379fbd371d55c76871 P 4e7aa6d387e94dffaff030eb416c66c4: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e7aa6d387e94dffaff030eb416c66c4" member_type: VOTER last_known_addr { host: "127.10.198.1" port: 41409 } } peers { permanent_uuid: "fe72d9fd358f4bf7967ac3ea2264aed9" member_type: VOTER last_known_addr { host: "127.10.198.2" port: 41061 } } peers { permanent_uuid: "7d80f46d2eac41e7a9f0b2b2e1972abd" member_type: VOTER last_known_addr { host: "127.10.198.3" port: 37891 } } }
I20260925 22:44:58.926386 19071 tablet_replica.cc:442] T e4c98185d11946379fbd371d55c76871 P 4e7aa6d387e94dffaff030eb416c66c4: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 22:44:58.926656 19064 raft_consensus.cc:348] T e4c98185d11946379fbd371d55c76871 P 7d80f46d2eac41e7a9f0b2b2e1972abd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e7aa6d387e94dffaff030eb416c66c4" member_type: VOTER last_known_addr { host: "127.10.198.1" port: 41409 } } peers { permanent_uuid: "fe72d9fd358f4bf7967ac3ea2264aed9" member_type: VOTER last_known_addr { host: "127.10.198.2" port: 41061 } } peers { permanent_uuid: "7d80f46d2eac41e7a9f0b2b2e1972abd" member_type: VOTER last_known_addr { host: "127.10.198.3" port: 37891 } }
I20260925 22:44:58.926774 19064 raft_consensus.cc:374] T e4c98185d11946379fbd371d55c76871 P 7d80f46d2eac41e7a9f0b2b2e1972abd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:44:58.926813 19064 raft_consensus.cc:729] T e4c98185d11946379fbd371d55c76871 P 7d80f46d2eac41e7a9f0b2b2e1972abd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7d80f46d2eac41e7a9f0b2b2e1972abd, State: Initialized, Role: FOLLOWER
I20260925 22:44:58.926949 19064 consensus_queue.cc:260] T e4c98185d11946379fbd371d55c76871 P 7d80f46d2eac41e7a9f0b2b2e1972abd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e7aa6d387e94dffaff030eb416c66c4" member_type: VOTER last_known_addr { host: "127.10.198.1" port: 41409 } } peers { permanent_uuid: "fe72d9fd358f4bf7967ac3ea2264aed9" member_type: VOTER last_known_addr { host: "127.10.198.2" port: 41061 } } peers { permanent_uuid: "7d80f46d2eac41e7a9f0b2b2e1972abd" member_type: VOTER last_known_addr { host: "127.10.198.3" port: 37891 } }
I20260925 22:44:58.927255 19064 ts_tablet_manager.cc:1428] T e4c98185d11946379fbd371d55c76871 P 7d80f46d2eac41e7a9f0b2b2e1972abd: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260925 22:44:58.927328 19070 tablet_replica.cc:439] T e4c98185d11946379fbd371d55c76871 P 7d80f46d2eac41e7a9f0b2b2e1972abd: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e7aa6d387e94dffaff030eb416c66c4" member_type: VOTER last_known_addr { host: "127.10.198.1" port: 41409 } } peers { permanent_uuid: "fe72d9fd358f4bf7967ac3ea2264aed9" member_type: VOTER last_known_addr { host: "127.10.198.2" port: 41061 } } peers { permanent_uuid: "7d80f46d2eac41e7a9f0b2b2e1972abd" member_type: VOTER last_known_addr { host: "127.10.198.3" port: 37891 } } }
I20260925 22:44:58.927469 19070 tablet_replica.cc:442] T e4c98185d11946379fbd371d55c76871 P 7d80f46d2eac41e7a9f0b2b2e1972abd: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 22:44:59.006834 19083 raft_consensus.cc:482] T e4c98185d11946379fbd371d55c76871 P fe72d9fd358f4bf7967ac3ea2264aed9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 22:44:59.007027 19083 raft_consensus.cc:504] T e4c98185d11946379fbd371d55c76871 P fe72d9fd358f4bf7967ac3ea2264aed9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e7aa6d387e94dffaff030eb416c66c4" member_type: VOTER last_known_addr { host: "127.10.198.1" port: 41409 } } peers { permanent_uuid: "fe72d9fd358f4bf7967ac3ea2264aed9" member_type: VOTER last_known_addr { host: "127.10.198.2" port: 41061 } } peers { permanent_uuid: "7d80f46d2eac41e7a9f0b2b2e1972abd" member_type: VOTER last_known_addr { host: "127.10.198.3" port: 37891 } }
I20260925 22:44:59.007687 19083 leader_election.cc:288] T e4c98185d11946379fbd371d55c76871 P fe72d9fd358f4bf7967ac3ea2264aed9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4e7aa6d387e94dffaff030eb416c66c4 (127.10.198.1:41409), 7d80f46d2eac41e7a9f0b2b2e1972abd (127.10.198.3:37891)
I20260925 22:44:59.008271 18705 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4c98185d11946379fbd371d55c76871" candidate_uuid: "fe72d9fd358f4bf7967ac3ea2264aed9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4e7aa6d387e94dffaff030eb416c66c4" is_pre_election: true
I20260925 22:44:59.008319 18990 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4c98185d11946379fbd371d55c76871" candidate_uuid: "fe72d9fd358f4bf7967ac3ea2264aed9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7d80f46d2eac41e7a9f0b2b2e1972abd" is_pre_election: true
I20260925 22:44:59.008455 18705 raft_consensus.cc:2454] T e4c98185d11946379fbd371d55c76871 P 4e7aa6d387e94dffaff030eb416c66c4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fe72d9fd358f4bf7967ac3ea2264aed9 in term 0.
I20260925 22:44:59.008523 18990 raft_consensus.cc:2454] T e4c98185d11946379fbd371d55c76871 P 7d80f46d2eac41e7a9f0b2b2e1972abd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fe72d9fd358f4bf7967ac3ea2264aed9 in term 0.
I20260925 22:44:59.008890 18783 leader_election.cc:302] T e4c98185d11946379fbd371d55c76871 P fe72d9fd358f4bf7967ac3ea2264aed9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4e7aa6d387e94dffaff030eb416c66c4, fe72d9fd358f4bf7967ac3ea2264aed9; no voters:
I20260925 22:44:59.009152 19083 raft_consensus.cc:2781] T e4c98185d11946379fbd371d55c76871 P fe72d9fd358f4bf7967ac3ea2264aed9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 22:44:59.009235 19083 raft_consensus.cc:482] T e4c98185d11946379fbd371d55c76871 P fe72d9fd358f4bf7967ac3ea2264aed9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 22:44:59.009292 19083 raft_consensus.cc:3037] T e4c98185d11946379fbd371d55c76871 P fe72d9fd358f4bf7967ac3ea2264aed9 [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:44:59.010383 19083 raft_consensus.cc:504] T e4c98185d11946379fbd371d55c76871 P fe72d9fd358f4bf7967ac3ea2264aed9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e7aa6d387e94dffaff030eb416c66c4" member_type: VOTER last_known_addr { host: "127.10.198.1" port: 41409 } } peers { permanent_uuid: "fe72d9fd358f4bf7967ac3ea2264aed9" member_type: VOTER last_known_addr { host: "127.10.198.2" port: 41061 } } peers { permanent_uuid: "7d80f46d2eac41e7a9f0b2b2e1972abd" member_type: VOTER last_known_addr { host: "127.10.198.3" port: 37891 } }
I20260925 22:44:59.010836 19083 leader_election.cc:288] T e4c98185d11946379fbd371d55c76871 P fe72d9fd358f4bf7967ac3ea2264aed9 [CANDIDATE]: Term 1 election: Requested vote from peers 4e7aa6d387e94dffaff030eb416c66c4 (127.10.198.1:41409), 7d80f46d2eac41e7a9f0b2b2e1972abd (127.10.198.3:37891)
I20260925 22:44:59.011296 18990 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4c98185d11946379fbd371d55c76871" candidate_uuid: "fe72d9fd358f4bf7967ac3ea2264aed9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7d80f46d2eac41e7a9f0b2b2e1972abd"
I20260925 22:44:59.011323 18705 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4c98185d11946379fbd371d55c76871" candidate_uuid: "fe72d9fd358f4bf7967ac3ea2264aed9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4e7aa6d387e94dffaff030eb416c66c4"
I20260925 22:44:59.011463 18990 raft_consensus.cc:3037] T e4c98185d11946379fbd371d55c76871 P 7d80f46d2eac41e7a9f0b2b2e1972abd [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:44:59.011507 18705 raft_consensus.cc:3037] T e4c98185d11946379fbd371d55c76871 P 4e7aa6d387e94dffaff030eb416c66c4 [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:44:59.012625 18990 raft_consensus.cc:2454] T e4c98185d11946379fbd371d55c76871 P 7d80f46d2eac41e7a9f0b2b2e1972abd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fe72d9fd358f4bf7967ac3ea2264aed9 in term 1.
I20260925 22:44:59.013098 18784 leader_election.cc:302] T e4c98185d11946379fbd371d55c76871 P fe72d9fd358f4bf7967ac3ea2264aed9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7d80f46d2eac41e7a9f0b2b2e1972abd, fe72d9fd358f4bf7967ac3ea2264aed9; no voters:
I20260925 22:44:59.013134 18705 raft_consensus.cc:2454] T e4c98185d11946379fbd371d55c76871 P 4e7aa6d387e94dffaff030eb416c66c4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fe72d9fd358f4bf7967ac3ea2264aed9 in term 1.
I20260925 22:44:59.013335 19083 raft_consensus.cc:2781] T e4c98185d11946379fbd371d55c76871 P fe72d9fd358f4bf7967ac3ea2264aed9 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 22:44:59.013599 19083 raft_consensus.cc:686] T e4c98185d11946379fbd371d55c76871 P fe72d9fd358f4bf7967ac3ea2264aed9 [term 1 LEADER]: Becoming Leader. State: Replica: fe72d9fd358f4bf7967ac3ea2264aed9, State: Running, Role: LEADER
I20260925 22:44:59.013762 19083 consensus_queue.cc:237] T e4c98185d11946379fbd371d55c76871 P fe72d9fd358f4bf7967ac3ea2264aed9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e7aa6d387e94dffaff030eb416c66c4" member_type: VOTER last_known_addr { host: "127.10.198.1" port: 41409 } } peers { permanent_uuid: "fe72d9fd358f4bf7967ac3ea2264aed9" member_type: VOTER last_known_addr { host: "127.10.198.2" port: 41061 } } peers { permanent_uuid: "7d80f46d2eac41e7a9f0b2b2e1972abd" member_type: VOTER last_known_addr { host: "127.10.198.3" port: 37891 } }
I20260925 22:44:59.014952 19075 tablet_replica.cc:439] T e4c98185d11946379fbd371d55c76871 P fe72d9fd358f4bf7967ac3ea2264aed9: TxnStatusTablet state changed. Reason: New leader fe72d9fd358f4bf7967ac3ea2264aed9. Latest consensus state: current_term: 1 leader_uuid: "fe72d9fd358f4bf7967ac3ea2264aed9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e7aa6d387e94dffaff030eb416c66c4" member_type: VOTER last_known_addr { host: "127.10.198.1" port: 41409 } } peers { permanent_uuid: "fe72d9fd358f4bf7967ac3ea2264aed9" member_type: VOTER last_known_addr { host: "127.10.198.2" port: 41061 } } peers { permanent_uuid: "7d80f46d2eac41e7a9f0b2b2e1972abd" member_type: VOTER last_known_addr { host: "127.10.198.3" port: 37891 } } }
I20260925 22:44:59.015128 19075 tablet_replica.cc:442] T e4c98185d11946379fbd371d55c76871 P fe72d9fd358f4bf7967ac3ea2264aed9: This TxnStatusTablet replica's current role is: LEADER
I20260925 22:44:59.015484 19109 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260925 22:44:59.015691 19109 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260925 22:44:59.016952 18552 catalog_manager.cc:5495] T e4c98185d11946379fbd371d55c76871 P fe72d9fd358f4bf7967ac3ea2264aed9 reported cstate change: term changed from 0 to 1, leader changed from <none> to fe72d9fd358f4bf7967ac3ea2264aed9 (127.10.198.2). New cstate: current_term: 1 leader_uuid: "fe72d9fd358f4bf7967ac3ea2264aed9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e7aa6d387e94dffaff030eb416c66c4" member_type: VOTER last_known_addr { host: "127.10.198.1" port: 41409 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fe72d9fd358f4bf7967ac3ea2264aed9" member_type: VOTER last_known_addr { host: "127.10.198.2" port: 41061 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7d80f46d2eac41e7a9f0b2b2e1972abd" member_type: VOTER last_known_addr { host: "127.10.198.3" port: 37891 } health_report { overall_health: UNKNOWN } } }
I20260925 22:44:59.050798 19103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:44:59.047255 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.198.2:41061 (local address 127.0.0.1:49958)
0925 22:44:59.047396 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0925 22:44:59.047414 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0925 22:44:59.047548 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:44:59.047910 (+ 362us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:44:59.047913 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:44:59.047921 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 22:44:59.048038 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:44:59.048043 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:44:59.048844 (+ 801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:44:59.048849 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:44:59.049471 (+ 622us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:44:59.049479 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:44:59.049979 (+ 500us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:44:59.049982 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:44:59.050076 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:44:59.050136 (+ 60us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 22:44:59.050470 (+ 334us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 22:44:59.050489 (+ 19us) client_negotiation.cc:769] Sending connection context
0925 22:44:59.050565 (+ 76us) client_negotiation.cc:240] Negotiation successful
0925 22:44:59.050664 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":29}
I20260925 22:44:59.050798 19062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:44:59.047500 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49958 (local address 127.10.198.2:41061)
0925 22:44:59.047636 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0925 22:44:59.047640 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:44:59.047656 (+ 16us) server_negotiation.cc:407] Connection header received
0925 22:44:59.047707 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:44:59.047710 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:44:59.047762 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:44:59.047851 (+ 89us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 22:44:59.048172 (+ 321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:44:59.048722 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:44:59.049611 (+ 889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:44:59.049857 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:44:59.049975 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:44:59.050281 (+ 306us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 22:44:59.050374 (+ 93us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 22:44:59.050456 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0925 22:44:59.050582 (+ 126us) server_negotiation.cc:299] Negotiation successful
0925 22:44:59.050642 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260925 22:44:59.053906 18990 raft_consensus.cc:1261] T e4c98185d11946379fbd371d55c76871 P 7d80f46d2eac41e7a9f0b2b2e1972abd [term 1 FOLLOWER]: Refusing update from remote peer fe72d9fd358f4bf7967ac3ea2264aed9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 22:44:59.053978 18705 raft_consensus.cc:1261] T e4c98185d11946379fbd371d55c76871 P 4e7aa6d387e94dffaff030eb416c66c4 [term 1 FOLLOWER]: Refusing update from remote peer fe72d9fd358f4bf7967ac3ea2264aed9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 22:44:59.054483 19083 consensus_queue.cc:1035] T e4c98185d11946379fbd371d55c76871 P fe72d9fd358f4bf7967ac3ea2264aed9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7d80f46d2eac41e7a9f0b2b2e1972abd" member_type: VOTER last_known_addr { host: "127.10.198.3" port: 37891 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 22:44:59.054667 19075 consensus_queue.cc:1035] T e4c98185d11946379fbd371d55c76871 P fe72d9fd358f4bf7967ac3ea2264aed9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e7aa6d387e94dffaff030eb416c66c4" member_type: VOTER last_known_addr { host: "127.10.198.1" port: 41409 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 22:44:59.056347 19071 tablet_replica.cc:439] T e4c98185d11946379fbd371d55c76871 P 4e7aa6d387e94dffaff030eb416c66c4: TxnStatusTablet state changed. Reason: New leader fe72d9fd358f4bf7967ac3ea2264aed9. Latest consensus state: current_term: 1 leader_uuid: "fe72d9fd358f4bf7967ac3ea2264aed9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e7aa6d387e94dffaff030eb416c66c4" member_type: VOTER last_known_addr { host: "127.10.198.1" port: 41409 } } peers { permanent_uuid: "fe72d9fd358f4bf7967ac3ea2264aed9" member_type: VOTER last_known_addr { host: "127.10.198.2" port: 41061 } } peers { permanent_uuid: "7d80f46d2eac41e7a9f0b2b2e1972abd" member_type: VOTER last_known_addr { host: "127.10.198.3" port: 37891 } } }
I20260925 22:44:59.056507 19071 tablet_replica.cc:442] T e4c98185d11946379fbd371d55c76871 P 4e7aa6d387e94dffaff030eb416c66c4: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 22:44:59.057286 19070 tablet_replica.cc:439] T e4c98185d11946379fbd371d55c76871 P 7d80f46d2eac41e7a9f0b2b2e1972abd: TxnStatusTablet state changed. Reason: New leader fe72d9fd358f4bf7967ac3ea2264aed9. Latest consensus state: current_term: 1 leader_uuid: "fe72d9fd358f4bf7967ac3ea2264aed9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e7aa6d387e94dffaff030eb416c66c4" member_type: VOTER last_known_addr { host: "127.10.198.1" port: 41409 } } peers { permanent_uuid: "fe72d9fd358f4bf7967ac3ea2264aed9" member_type: VOTER last_known_addr { host: "127.10.198.2" port: 41061 } } peers { permanent_uuid: "7d80f46d2eac41e7a9f0b2b2e1972abd" member_type: VOTER last_known_addr { host: "127.10.198.3" port: 37891 } } }
I20260925 22:44:59.057410 19070 tablet_replica.cc:442] T e4c98185d11946379fbd371d55c76871 P 7d80f46d2eac41e7a9f0b2b2e1972abd: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 22:44:59.059918 19072 tablet_replica.cc:439] T e4c98185d11946379fbd371d55c76871 P fe72d9fd358f4bf7967ac3ea2264aed9: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "fe72d9fd358f4bf7967ac3ea2264aed9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e7aa6d387e94dffaff030eb416c66c4" member_type: VOTER last_known_addr { host: "127.10.198.1" port: 41409 } } peers { permanent_uuid: "fe72d9fd358f4bf7967ac3ea2264aed9" member_type: VOTER last_known_addr { host: "127.10.198.2" port: 41061 } } peers { permanent_uuid: "7d80f46d2eac41e7a9f0b2b2e1972abd" member_type: VOTER last_known_addr { host: "127.10.198.3" port: 37891 } } }
I20260925 22:44:59.060088 19072 tablet_replica.cc:442] T e4c98185d11946379fbd371d55c76871 P fe72d9fd358f4bf7967ac3ea2264aed9: This TxnStatusTablet replica's current role is: LEADER
I20260925 22:44:59.063643 19080 mvcc.cc:205] Tried to move back new op lower bound from 7333381320919523328 to 7333381320762077184. Current Snapshot: MvccSnapshot[applied={T|T < 7333381320919523328}]
I20260925 22:44:59.063898 19070 tablet_replica.cc:439] T e4c98185d11946379fbd371d55c76871 P 7d80f46d2eac41e7a9f0b2b2e1972abd: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "fe72d9fd358f4bf7967ac3ea2264aed9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e7aa6d387e94dffaff030eb416c66c4" member_type: VOTER last_known_addr { host: "127.10.198.1" port: 41409 } } peers { permanent_uuid: "fe72d9fd358f4bf7967ac3ea2264aed9" member_type: VOTER last_known_addr { host: "127.10.198.2" port: 41061 } } peers { permanent_uuid: "7d80f46d2eac41e7a9f0b2b2e1972abd" member_type: VOTER last_known_addr { host: "127.10.198.3" port: 37891 } } }
I20260925 22:44:59.064039 19070 tablet_replica.cc:442] T e4c98185d11946379fbd371d55c76871 P 7d80f46d2eac41e7a9f0b2b2e1972abd: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 22:44:59.065922 19083 tablet_replica.cc:439] T e4c98185d11946379fbd371d55c76871 P fe72d9fd358f4bf7967ac3ea2264aed9: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "fe72d9fd358f4bf7967ac3ea2264aed9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e7aa6d387e94dffaff030eb416c66c4" member_type: VOTER last_known_addr { host: "127.10.198.1" port: 41409 } } peers { permanent_uuid: "fe72d9fd358f4bf7967ac3ea2264aed9" member_type: VOTER last_known_addr { host: "127.10.198.2" port: 41061 } } peers { permanent_uuid: "7d80f46d2eac41e7a9f0b2b2e1972abd" member_type: VOTER last_known_addr { host: "127.10.198.3" port: 37891 } } }
I20260925 22:44:59.066097 19083 tablet_replica.cc:442] T e4c98185d11946379fbd371d55c76871 P fe72d9fd358f4bf7967ac3ea2264aed9: This TxnStatusTablet replica's current role is: LEADER
I20260925 22:44:59.067065 19071 tablet_replica.cc:439] T e4c98185d11946379fbd371d55c76871 P 4e7aa6d387e94dffaff030eb416c66c4: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "fe72d9fd358f4bf7967ac3ea2264aed9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e7aa6d387e94dffaff030eb416c66c4" member_type: VOTER last_known_addr { host: "127.10.198.1" port: 41409 } } peers { permanent_uuid: "fe72d9fd358f4bf7967ac3ea2264aed9" member_type: VOTER last_known_addr { host: "127.10.198.2" port: 41061 } } peers { permanent_uuid: "7d80f46d2eac41e7a9f0b2b2e1972abd" member_type: VOTER last_known_addr { host: "127.10.198.3" port: 37891 } } }
I20260925 22:44:59.067199 19071 tablet_replica.cc:442] T e4c98185d11946379fbd371d55c76871 P 4e7aa6d387e94dffaff030eb416c66c4: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 22:44:59.074613 19060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:44:59.069705 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39260 (local address 127.10.198.1:41409)
0925 22:44:59.069843 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0925 22:44:59.069849 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 22:44:59.069907 (+ 58us) server_negotiation.cc:407] Connection header received
0925 22:44:59.070122 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:44:59.070126 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:44:59.070198 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:44:59.070308 (+ 110us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 22:44:59.070735 (+ 427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:44:59.071423 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:44:59.072674 (+ 1251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:44:59.073011 (+ 337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:44:59.073114 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:44:59.073523 (+ 409us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 22:44:59.074181 (+ 658us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 22:44:59.074316 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0925 22:44:59.074363 (+ 47us) server_negotiation.cc:299] Negotiation successful
0925 22:44:59.074424 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260925 22:44:59.086673 19116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:44:59.082672 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.198.2:41061 (local address 127.10.198.1:38491)
0925 22:44:59.082976 (+ 304us) negotiation.cc:107] Waiting for socket to connect
0925 22:44:59.082993 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 22:44:59.083094 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:44:59.083380 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:44:59.083383 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:44:59.083391 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 22:44:59.083521 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:44:59.083527 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:44:59.084333 (+ 806us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:44:59.084339 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:44:59.085163 (+ 824us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:44:59.085174 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:44:59.085697 (+ 523us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:44:59.085701 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:44:59.085831 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:44:59.085893 (+ 62us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 22:44:59.086307 (+ 414us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 22:44:59.086313 (+ 6us) client_negotiation.cc:769] Sending connection context
0925 22:44:59.086408 (+ 95us) client_negotiation.cc:240] Negotiation successful
0925 22:44:59.086504 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":110,"threads_started":1}
I20260925 22:44:59.086659 19062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:44:59.082733 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.198.1:38491 (local address 127.10.198.2:41061)
0925 22:44:59.083000 (+ 267us) server_negotiation.cc:206] Beginning negotiation
0925 22:44:59.083006 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 22:44:59.083052 (+ 46us) server_negotiation.cc:407] Connection header received
0925 22:44:59.083219 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:44:59.083222 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:44:59.083281 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:44:59.083358 (+ 77us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 22:44:59.083630 (+ 272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:44:59.084224 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:44:59.085318 (+ 1094us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:44:59.085556 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:44:59.085690 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:44:59.086066 (+ 376us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 22:44:59.086181 (+ 115us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 22:44:59.086237 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0925 22:44:59.086426 (+ 189us) server_negotiation.cc:299] Negotiation successful
0925 22:44:59.086487 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":190}
I20260925 22:44:59.095299 18990 raft_consensus.cc:1261] T 8bea55fc27e746adb1fb1c2e7d4959e8 P 7d80f46d2eac41e7a9f0b2b2e1972abd [term 1 FOLLOWER]: Refusing update from remote peer 4e7aa6d387e94dffaff030eb416c66c4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 22:44:59.095810 19071 consensus_queue.cc:1035] T 8bea55fc27e746adb1fb1c2e7d4959e8 P 4e7aa6d387e94dffaff030eb416c66c4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7d80f46d2eac41e7a9f0b2b2e1972abd" member_type: VOTER last_known_addr { host: "127.10.198.3" port: 37891 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 22:44:59.096556 18850 raft_consensus.cc:1261] T 8bea55fc27e746adb1fb1c2e7d4959e8 P fe72d9fd358f4bf7967ac3ea2264aed9 [term 1 FOLLOWER]: Refusing update from remote peer 4e7aa6d387e94dffaff030eb416c66c4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 22:44:59.099479 19071 consensus_queue.cc:1035] T 8bea55fc27e746adb1fb1c2e7d4959e8 P 4e7aa6d387e94dffaff030eb416c66c4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fe72d9fd358f4bf7967ac3ea2264aed9" member_type: VOTER last_known_addr { host: "127.10.198.2" port: 41061 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 22:44:59.130941 19060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:44:59.126148 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.198.2:44367 (local address 127.10.198.1:41409)
0925 22:44:59.126293 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0925 22:44:59.126299 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 22:44:59.126682 (+ 383us) server_negotiation.cc:407] Connection header received
0925 22:44:59.126939 (+ 257us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:44:59.126943 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:44:59.127006 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:44:59.127064 (+ 58us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 22:44:59.127478 (+ 414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:44:59.128345 (+ 867us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:44:59.129478 (+ 1133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:44:59.129817 (+ 339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:44:59.129935 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:44:59.130396 (+ 461us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 22:44:59.130505 (+ 109us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 22:44:59.130559 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0925 22:44:59.130742 (+ 183us) server_negotiation.cc:299] Negotiation successful
0925 22:44:59.130821 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260925 22:44:59.131901 19121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:44:59.125707 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.198.1:41409 (local address 127.10.198.2:44367)
0925 22:44:59.126569 (+ 862us) negotiation.cc:107] Waiting for socket to connect
0925 22:44:59.126600 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0925 22:44:59.126787 (+ 187us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:44:59.127222 (+ 435us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:44:59.127225 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:44:59.127237 (+ 12us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 22:44:59.127340 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:44:59.127347 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:44:59.128494 (+ 1147us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:44:59.128497 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:44:59.129316 (+ 819us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:44:59.129327 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:44:59.130059 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:44:59.130063 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:44:59.130159 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:44:59.130223 (+ 64us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 22:44:59.130633 (+ 410us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 22:44:59.130639 (+ 6us) client_negotiation.cc:769] Sending connection context
0925 22:44:59.131583 (+ 944us) client_negotiation.cc:240] Negotiation successful
0925 22:44:59.131695 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":727,"thread_start_us":92,"threads_started":1}
I20260925 22:44:59.169968 18551 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55532:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 22:44:59.170182 18551 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55532:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 22:44:59.173048 18551 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9a42cb7f7adf47f1b0dd00bf3bfbd491: Sending DeleteTablet for 3 replicas of tablet 8bea55fc27e746adb1fb1c2e7d4959e8
I20260925 22:44:59.174239 18970 tablet_service.cc:1514] Processing DeleteTablet for tablet 8bea55fc27e746adb1fb1c2e7d4959e8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 22:44:59 UTC) from {username='oryx', principal='oryx/127.10.198.62@KRBTEST.COM'} at 127.0.0.1:60530
I20260925 22:44:59.174261 18683 tablet_service.cc:1514] Processing DeleteTablet for tablet 8bea55fc27e746adb1fb1c2e7d4959e8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 22:44:59 UTC) from {username='oryx', principal='oryx/127.10.198.62@KRBTEST.COM'} at 127.0.0.1:39244
I20260925 22:44:59.174494 19090 tablet_replica.cc:330] T 8bea55fc27e746adb1fb1c2e7d4959e8 P 7d80f46d2eac41e7a9f0b2b2e1972abd: stopping tablet replica
I20260925 22:44:59.174527 19088 tablet_replica.cc:330] T 8bea55fc27e746adb1fb1c2e7d4959e8 P 4e7aa6d387e94dffaff030eb416c66c4: stopping tablet replica
I20260925 22:44:59.174646 19088 raft_consensus.cc:2229] T 8bea55fc27e746adb1fb1c2e7d4959e8 P 4e7aa6d387e94dffaff030eb416c66c4 [term 1 LEADER]: Raft consensus shutting down.
I20260925 22:44:59.174646 19090 raft_consensus.cc:2229] T 8bea55fc27e746adb1fb1c2e7d4959e8 P 7d80f46d2eac41e7a9f0b2b2e1972abd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 22:44:59.174826 19090 raft_consensus.cc:2258] T 8bea55fc27e746adb1fb1c2e7d4959e8 P 7d80f46d2eac41e7a9f0b2b2e1972abd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 22:44:59.175020 19088 raft_consensus.cc:2258] T 8bea55fc27e746adb1fb1c2e7d4959e8 P 4e7aa6d387e94dffaff030eb416c66c4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 22:44:59.175920 19090 ts_tablet_manager.cc:1905] T 8bea55fc27e746adb1fb1c2e7d4959e8 P 7d80f46d2eac41e7a9f0b2b2e1972abd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 22:44:59.176209 18830 tablet_service.cc:1514] Processing DeleteTablet for tablet 8bea55fc27e746adb1fb1c2e7d4959e8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 22:44:59 UTC) from {username='oryx', principal='oryx/127.10.198.62@KRBTEST.COM'} at 127.0.0.1:49946
I20260925 22:44:59.176432 19089 tablet_replica.cc:330] T 8bea55fc27e746adb1fb1c2e7d4959e8 P fe72d9fd358f4bf7967ac3ea2264aed9: stopping tablet replica
I20260925 22:44:59.176551 19089 raft_consensus.cc:2229] T 8bea55fc27e746adb1fb1c2e7d4959e8 P fe72d9fd358f4bf7967ac3ea2264aed9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 22:44:59.176599 19088 ts_tablet_manager.cc:1905] T 8bea55fc27e746adb1fb1c2e7d4959e8 P 4e7aa6d387e94dffaff030eb416c66c4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 22:44:59.176669 19089 raft_consensus.cc:2258] T 8bea55fc27e746adb1fb1c2e7d4959e8 P fe72d9fd358f4bf7967ac3ea2264aed9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 22:44:59.177625 19089 ts_tablet_manager.cc:1905] T 8bea55fc27e746adb1fb1c2e7d4959e8 P fe72d9fd358f4bf7967ac3ea2264aed9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 22:44:59.179301 19090 ts_tablet_manager.cc:1918] T 8bea55fc27e746adb1fb1c2e7d4959e8 P 7d80f46d2eac41e7a9f0b2b2e1972abd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 22:44:59.179414 19090 log.cc:1196] T 8bea55fc27e746adb1fb1c2e7d4959e8 P 7d80f46d2eac41e7a9f0b2b2e1972abd: Deleting WAL directory at /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790376259650642-11032-0/minicluster-data/ts-2/wal/wals/8bea55fc27e746adb1fb1c2e7d4959e8
I20260925 22:44:59.179531 19088 ts_tablet_manager.cc:1918] T 8bea55fc27e746adb1fb1c2e7d4959e8 P 4e7aa6d387e94dffaff030eb416c66c4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 22:44:59.179618 19088 log.cc:1196] T 8bea55fc27e746adb1fb1c2e7d4959e8 P 4e7aa6d387e94dffaff030eb416c66c4: Deleting WAL directory at /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790376259650642-11032-0/minicluster-data/ts-0/wal/wals/8bea55fc27e746adb1fb1c2e7d4959e8
I20260925 22:44:59.179692 19090 ts_tablet_manager.cc:1939] T 8bea55fc27e746adb1fb1c2e7d4959e8 P 7d80f46d2eac41e7a9f0b2b2e1972abd: Deleting consensus metadata
I20260925 22:44:59.179904 19088 ts_tablet_manager.cc:1939] T 8bea55fc27e746adb1fb1c2e7d4959e8 P 4e7aa6d387e94dffaff030eb416c66c4: Deleting consensus metadata
I20260925 22:44:59.180567 19089 ts_tablet_manager.cc:1918] T 8bea55fc27e746adb1fb1c2e7d4959e8 P fe72d9fd358f4bf7967ac3ea2264aed9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 22:44:59.180668 19089 log.cc:1196] T 8bea55fc27e746adb1fb1c2e7d4959e8 P fe72d9fd358f4bf7967ac3ea2264aed9: Deleting WAL directory at /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790376259650642-11032-0/minicluster-data/ts-1/wal/wals/8bea55fc27e746adb1fb1c2e7d4959e8
I20260925 22:44:59.180770 18539 catalog_manager.cc:4841] TS 4e7aa6d387e94dffaff030eb416c66c4 (127.10.198.1:41409): tablet 8bea55fc27e746adb1fb1c2e7d4959e8 (table test-table [id=a0575eaaf0aa43158e30e3dcf5bb1564]) successfully deleted
I20260925 22:44:59.180971 19089 ts_tablet_manager.cc:1939] T 8bea55fc27e746adb1fb1c2e7d4959e8 P fe72d9fd358f4bf7967ac3ea2264aed9: Deleting consensus metadata
I20260925 22:44:59.181798 18540 catalog_manager.cc:4841] TS fe72d9fd358f4bf7967ac3ea2264aed9 (127.10.198.2:41061): tablet 8bea55fc27e746adb1fb1c2e7d4959e8 (table test-table [id=a0575eaaf0aa43158e30e3dcf5bb1564]) successfully deleted
I20260925 22:44:59.184064 18540 catalog_manager.cc:4841] TS 7d80f46d2eac41e7a9f0b2b2e1972abd (127.10.198.3:37891): tablet 8bea55fc27e746adb1fb1c2e7d4959e8 (table test-table [id=a0575eaaf0aa43158e30e3dcf5bb1564]) successfully deleted
Sep 25 22:44:59 dist-test-slave-5vdr krb5kdc[18508](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376299, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 25 22:44:59 dist-test-slave-5vdr krb5kdc[18508](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.10.198.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 22:44:59 dist-test-slave-5vdr krb5kdc[18508](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.10.198.62@KRBTEST.COM, Server not found in Kerberos database
W20260925 22:44:59.218412 19104 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:44:59.208639 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55550 (local address 127.10.198.62:46765)
0925 22:44:59.208780 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0925 22:44:59.208788 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 22:44:59.209032 (+ 244us) server_negotiation.cc:407] Connection header received
0925 22:44:59.209187 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:44:59.209191 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:44:59.209260 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:44:59.209373 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:44:59.210466 (+ 1093us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:44:59.211298 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:44:59.212881 (+ 1583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:44:59.213255 (+ 374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:44:59.213411 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:44:59.218251 (+ 4840us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55550: BlockingRecv error: recv got EOF from 127.0.0.1:55550 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
Sep 25 22:44:59 dist-test-slave-5vdr krb5kdc[18508](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376299, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.10.198.62@KRBTEST.COM
I20260925 22:44:59.234025 19104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:44:59.220969 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55556 (local address 127.10.198.62:46765)
0925 22:44:59.221112 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0925 22:44:59.221117 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:44:59.221415 (+ 298us) server_negotiation.cc:407] Connection header received
0925 22:44:59.221559 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:44:59.221563 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:44:59.221632 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:44:59.221749 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:44:59.222714 (+ 965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:44:59.223541 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:44:59.224962 (+ 1421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:44:59.225299 (+ 337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:44:59.225451 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:44:59.229201 (+ 3750us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:44:59.229207 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:44:59.229211 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:44:59.229259 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:44:59.231529 (+ 2270us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:44:59.232344 (+ 815us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:44:59.232349 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:44:59.232351 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:44:59.232478 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:44:59.232864 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:44:59.232868 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:44:59.232870 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:44:59.233143 (+ 273us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:44:59.233295 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0925 22:44:59.233663 (+ 368us) server_negotiation.cc:299] Negotiation successful
0925 22:44:59.233830 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260925 22:44:59.237957 18551 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55556:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 22:44:59.238538 18551 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 22:44:59.245393 18830 tablet_service.cc:1467] Processing CreateTablet for tablet 2069a83262f9446ca244a563aa9ad319 (DEFAULT_TABLE table=test-table [id=899ab479aa0e4676b6bd01c1931e0f07]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 22:44:59.245393 18970 tablet_service.cc:1467] Processing CreateTablet for tablet 2069a83262f9446ca244a563aa9ad319 (DEFAULT_TABLE table=test-table [id=899ab479aa0e4676b6bd01c1931e0f07]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 22:44:59.245755 18830 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2069a83262f9446ca244a563aa9ad319. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:44:59.245755 18970 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2069a83262f9446ca244a563aa9ad319. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:44:59.248628 19066 tablet_bootstrap.cc:492] T 2069a83262f9446ca244a563aa9ad319 P fe72d9fd358f4bf7967ac3ea2264aed9: Bootstrap starting.
I20260925 22:44:59.249557 18683 tablet_service.cc:1467] Processing CreateTablet for tablet 2069a83262f9446ca244a563aa9ad319 (DEFAULT_TABLE table=test-table [id=899ab479aa0e4676b6bd01c1931e0f07]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 22:44:59.249686 19066 tablet_bootstrap.cc:654] T 2069a83262f9446ca244a563aa9ad319 P fe72d9fd358f4bf7967ac3ea2264aed9: Neither blocks nor log segments found. Creating new log.
I20260925 22:44:59.249877 18683 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2069a83262f9446ca244a563aa9ad319. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:44:59.251219 19066 tablet_bootstrap.cc:492] T 2069a83262f9446ca244a563aa9ad319 P fe72d9fd358f4bf7967ac3ea2264aed9: No bootstrap required, opened a new log
I20260925 22:44:59.251318 19066 ts_tablet_manager.cc:1397] T 2069a83262f9446ca244a563aa9ad319 P fe72d9fd358f4bf7967ac3ea2264aed9: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260925 22:44:59.251783 19066 raft_consensus.cc:348] T 2069a83262f9446ca244a563aa9ad319 P fe72d9fd358f4bf7967ac3ea2264aed9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e7aa6d387e94dffaff030eb416c66c4" member_type: VOTER last_known_addr { host: "127.10.198.1" port: 41409 } } peers { permanent_uuid: "fe72d9fd358f4bf7967ac3ea2264aed9" member_type: VOTER last_known_addr { host: "127.10.198.2" port: 41061 } } peers { permanent_uuid: "7d80f46d2eac41e7a9f0b2b2e1972abd" member_type: VOTER last_known_addr { host: "127.10.198.3" port: 37891 } }
I20260925 22:44:59.251942 19066 raft_consensus.cc:374] T 2069a83262f9446ca244a563aa9ad319 P fe72d9fd358f4bf7967ac3ea2264aed9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:44:59.251977 19066 raft_consensus.cc:729] T 2069a83262f9446ca244a563aa9ad319 P fe72d9fd358f4bf7967ac3ea2264aed9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe72d9fd358f4bf7967ac3ea2264aed9, State: Initialized, Role: FOLLOWER
I20260925 22:44:59.252136 19066 consensus_queue.cc:260] T 2069a83262f9446ca244a563aa9ad319 P fe72d9fd358f4bf7967ac3ea2264aed9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e7aa6d387e94dffaff030eb416c66c4" member_type: VOTER last_known_addr { host: "127.10.198.1" port: 41409 } } peers { permanent_uuid: "fe72d9fd358f4bf7967ac3ea2264aed9" member_type: VOTER last_known_addr { host: "127.10.198.2" port: 41061 } } peers { permanent_uuid: "7d80f46d2eac41e7a9f0b2b2e1972abd" member_type: VOTER last_known_addr { host: "127.10.198.3" port: 37891 } }
I20260925 22:44:59.252439 19066 ts_tablet_manager.cc:1428] T 2069a83262f9446ca244a563aa9ad319 P fe72d9fd358f4bf7967ac3ea2264aed9: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260925 22:44:59.254104 19065 tablet_bootstrap.cc:492] T 2069a83262f9446ca244a563aa9ad319 P 4e7aa6d387e94dffaff030eb416c66c4: Bootstrap starting.
I20260925 22:44:59.255308 19065 tablet_bootstrap.cc:654] T 2069a83262f9446ca244a563aa9ad319 P 4e7aa6d387e94dffaff030eb416c66c4: Neither blocks nor log segments found. Creating new log.
I20260925 22:44:59.256651 19064 tablet_bootstrap.cc:492] T 2069a83262f9446ca244a563aa9ad319 P 7d80f46d2eac41e7a9f0b2b2e1972abd: Bootstrap starting.
I20260925 22:44:59.257733 19064 tablet_bootstrap.cc:654] T 2069a83262f9446ca244a563aa9ad319 P 7d80f46d2eac41e7a9f0b2b2e1972abd: Neither blocks nor log segments found. Creating new log.
I20260925 22:44:59.259321 19064 tablet_bootstrap.cc:492] T 2069a83262f9446ca244a563aa9ad319 P 7d80f46d2eac41e7a9f0b2b2e1972abd: No bootstrap required, opened a new log
I20260925 22:44:59.259447 19064 ts_tablet_manager.cc:1397] T 2069a83262f9446ca244a563aa9ad319 P 7d80f46d2eac41e7a9f0b2b2e1972abd: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260925 22:44:59.260085 19064 raft_consensus.cc:348] T 2069a83262f9446ca244a563aa9ad319 P 7d80f46d2eac41e7a9f0b2b2e1972abd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e7aa6d387e94dffaff030eb416c66c4" member_type: VOTER last_known_addr { host: "127.10.198.1" port: 41409 } } peers { permanent_uuid: "fe72d9fd358f4bf7967ac3ea2264aed9" member_type: VOTER last_known_addr { host: "127.10.198.2" port: 41061 } } peers { permanent_uuid: "7d80f46d2eac41e7a9f0b2b2e1972abd" member_type: VOTER last_known_addr { host: "127.10.198.3" port: 37891 } }
I20260925 22:44:59.260241 19064 raft_consensus.cc:374] T 2069a83262f9446ca244a563aa9ad319 P 7d80f46d2eac41e7a9f0b2b2e1972abd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:44:59.260277 19064 raft_consensus.cc:729] T 2069a83262f9446ca244a563aa9ad319 P 7d80f46d2eac41e7a9f0b2b2e1972abd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7d80f46d2eac41e7a9f0b2b2e1972abd, State: Initialized, Role: FOLLOWER
I20260925 22:44:59.260462 19064 consensus_queue.cc:260] T 2069a83262f9446ca244a563aa9ad319 P 7d80f46d2eac41e7a9f0b2b2e1972abd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e7aa6d387e94dffaff030eb416c66c4" member_type: VOTER last_known_addr { host: "127.10.198.1" port: 41409 } } peers { permanent_uuid: "fe72d9fd358f4bf7967ac3ea2264aed9" member_type: VOTER last_known_addr { host: "127.10.198.2" port: 41061 } } peers { permanent_uuid: "7d80f46d2eac41e7a9f0b2b2e1972abd" member_type: VOTER last_known_addr { host: "127.10.198.3" port: 37891 } }
I20260925 22:44:59.261355 19064 ts_tablet_manager.cc:1428] T 2069a83262f9446ca244a563aa9ad319 P 7d80f46d2eac41e7a9f0b2b2e1972abd: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260925 22:44:59.265300 19065 tablet_bootstrap.cc:492] T 2069a83262f9446ca244a563aa9ad319 P 4e7aa6d387e94dffaff030eb416c66c4: No bootstrap required, opened a new log
I20260925 22:44:59.265451 19065 ts_tablet_manager.cc:1397] T 2069a83262f9446ca244a563aa9ad319 P 4e7aa6d387e94dffaff030eb416c66c4: Time spent bootstrapping tablet: real 0.011s user 0.002s sys 0.000s
I20260925 22:44:59.266026 19065 raft_consensus.cc:348] T 2069a83262f9446ca244a563aa9ad319 P 4e7aa6d387e94dffaff030eb416c66c4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e7aa6d387e94dffaff030eb416c66c4" member_type: VOTER last_known_addr { host: "127.10.198.1" port: 41409 } } peers { permanent_uuid: "fe72d9fd358f4bf7967ac3ea2264aed9" member_type: VOTER last_known_addr { host: "127.10.198.2" port: 41061 } } peers { permanent_uuid: "7d80f46d2eac41e7a9f0b2b2e1972abd" member_type: VOTER last_known_addr { host: "127.10.198.3" port: 37891 } }
I20260925 22:44:59.266180 19065 raft_consensus.cc:374] T 2069a83262f9446ca244a563aa9ad319 P 4e7aa6d387e94dffaff030eb416c66c4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:44:59.266213 19065 raft_consensus.cc:729] T 2069a83262f9446ca244a563aa9ad319 P 4e7aa6d387e94dffaff030eb416c66c4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4e7aa6d387e94dffaff030eb416c66c4, State: Initialized, Role: FOLLOWER
I20260925 22:44:59.266741 19065 consensus_queue.cc:260] T 2069a83262f9446ca244a563aa9ad319 P 4e7aa6d387e94dffaff030eb416c66c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e7aa6d387e94dffaff030eb416c66c4" member_type: VOTER last_known_addr { host: "127.10.198.1" port: 41409 } } peers { permanent_uuid: "fe72d9fd358f4bf7967ac3ea2264aed9" member_type: VOTER last_known_addr { host: "127.10.198.2" port: 41061 } } peers { permanent_uuid: "7d80f46d2eac41e7a9f0b2b2e1972abd" member_type: VOTER last_known_addr { host: "127.10.198.3" port: 37891 } }
I20260925 22:44:59.267656 19065 ts_tablet_manager.cc:1428] T 2069a83262f9446ca244a563aa9ad319 P 4e7aa6d387e94dffaff030eb416c66c4: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260925 22:44:59.347106 19070 raft_consensus.cc:482] T 2069a83262f9446ca244a563aa9ad319 P 7d80f46d2eac41e7a9f0b2b2e1972abd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 22:44:59.347289 19070 raft_consensus.cc:504] T 2069a83262f9446ca244a563aa9ad319 P 7d80f46d2eac41e7a9f0b2b2e1972abd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e7aa6d387e94dffaff030eb416c66c4" member_type: VOTER last_known_addr { host: "127.10.198.1" port: 41409 } } peers { permanent_uuid: "fe72d9fd358f4bf7967ac3ea2264aed9" member_type: VOTER last_known_addr { host: "127.10.198.2" port: 41061 } } peers { permanent_uuid: "7d80f46d2eac41e7a9f0b2b2e1972abd" member_type: VOTER last_known_addr { host: "127.10.198.3" port: 37891 } }
I20260925 22:44:59.348917 19070 leader_election.cc:288] T 2069a83262f9446ca244a563aa9ad319 P 7d80f46d2eac41e7a9f0b2b2e1972abd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4e7aa6d387e94dffaff030eb416c66c4 (127.10.198.1:41409), fe72d9fd358f4bf7967ac3ea2264aed9 (127.10.198.2:41061)
I20260925 22:44:59.354364 19060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:44:59.349306 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.198.3:51977 (local address 127.10.198.1:41409)
0925 22:44:59.349568 (+ 262us) server_negotiation.cc:206] Beginning negotiation
0925 22:44:59.349575 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 22:44:59.349600 (+ 25us) server_negotiation.cc:407] Connection header received
0925 22:44:59.349815 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:44:59.349819 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:44:59.349893 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:44:59.349958 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:44:59.350529 (+ 571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:44:59.351347 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:44:59.353102 (+ 1755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:44:59.353562 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:44:59.353744 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:44:59.353814 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0925 22:44:59.354067 (+ 253us) server_negotiation.cc:299] Negotiation successful
0925 22:44:59.354180 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":172}
I20260925 22:44:59.354533 19062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:44:59.349798 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.198.3:52203 (local address 127.10.198.2:41061)
0925 22:44:59.349931 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0925 22:44:59.349936 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:44:59.349953 (+ 17us) server_negotiation.cc:407] Connection header received
0925 22:44:59.350013 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:44:59.350016 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:44:59.350069 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:44:59.350171 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:44:59.350561 (+ 390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:44:59.351103 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:44:59.353081 (+ 1978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:44:59.353819 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:44:59.354208 (+ 389us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:44:59.354268 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0925 22:44:59.354312 (+ 44us) server_negotiation.cc:299] Negotiation successful
0925 22:44:59.354390 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260925 22:44:59.354565 19140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:44:59.349403 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.198.2:41061 (local address 127.10.198.3:52203)
0925 22:44:59.349740 (+ 337us) negotiation.cc:107] Waiting for socket to connect
0925 22:44:59.349757 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 22:44:59.349829 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:44:59.350192 (+ 363us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:44:59.350195 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:44:59.350208 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:44:59.350392 (+ 184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:44:59.350402 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:44:59.351251 (+ 849us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:44:59.351258 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:44:59.352831 (+ 1573us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:44:59.352852 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:44:59.354001 (+ 1149us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:44:59.354009 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:44:59.354104 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:44:59.354128 (+ 24us) client_negotiation.cc:769] Sending connection context
0925 22:44:59.354199 (+ 71us) client_negotiation.cc:240] Negotiation successful
0925 22:44:59.354280 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":267,"mutex_wait_us":147,"thread_start_us":72,"threads_started":1}
I20260925 22:44:59.354944 19139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:44:59.349220 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.198.1:41409 (local address 127.10.198.3:51977)
0925 22:44:59.349513 (+ 293us) negotiation.cc:107] Waiting for socket to connect
0925 22:44:59.349533 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0925 22:44:59.349663 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:44:59.350093 (+ 430us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:44:59.350095 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:44:59.350110 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:44:59.350302 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:44:59.350369 (+ 67us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:44:59.351504 (+ 1135us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:44:59.351508 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:44:59.352831 (+ 1323us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:44:59.352852 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:44:59.353744 (+ 892us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:44:59.353749 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:44:59.353881 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:44:59.353917 (+ 36us) client_negotiation.cc:769] Sending connection context
0925 22:44:59.354677 (+ 760us) client_negotiation.cc:240] Negotiation successful
0925 22:44:59.354780 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":200,"mutex_wait_us":34,"thread_start_us":78,"threads_started":1}
I20260925 22:44:59.355423 18850 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2069a83262f9446ca244a563aa9ad319" candidate_uuid: "7d80f46d2eac41e7a9f0b2b2e1972abd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe72d9fd358f4bf7967ac3ea2264aed9" is_pre_election: true
I20260925 22:44:59.355427 18705 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2069a83262f9446ca244a563aa9ad319" candidate_uuid: "7d80f46d2eac41e7a9f0b2b2e1972abd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4e7aa6d387e94dffaff030eb416c66c4" is_pre_election: true
I20260925 22:44:59.355624 18705 raft_consensus.cc:2454] T 2069a83262f9446ca244a563aa9ad319 P 4e7aa6d387e94dffaff030eb416c66c4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7d80f46d2eac41e7a9f0b2b2e1972abd in term 0.
I20260925 22:44:59.355625 18850 raft_consensus.cc:2454] T 2069a83262f9446ca244a563aa9ad319 P fe72d9fd358f4bf7967ac3ea2264aed9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7d80f46d2eac41e7a9f0b2b2e1972abd in term 0.
I20260925 22:44:59.356119 18923 leader_election.cc:302] T 2069a83262f9446ca244a563aa9ad319 P 7d80f46d2eac41e7a9f0b2b2e1972abd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 4e7aa6d387e94dffaff030eb416c66c4, 7d80f46d2eac41e7a9f0b2b2e1972abd, fe72d9fd358f4bf7967ac3ea2264aed9; no voters:
I20260925 22:44:59.356467 19070 raft_consensus.cc:2781] T 2069a83262f9446ca244a563aa9ad319 P 7d80f46d2eac41e7a9f0b2b2e1972abd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 22:44:59.356565 19070 raft_consensus.cc:482] T 2069a83262f9446ca244a563aa9ad319 P 7d80f46d2eac41e7a9f0b2b2e1972abd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 22:44:59.356622 19070 raft_consensus.cc:3037] T 2069a83262f9446ca244a563aa9ad319 P 7d80f46d2eac41e7a9f0b2b2e1972abd [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:44:59.357693 19070 raft_consensus.cc:504] T 2069a83262f9446ca244a563aa9ad319 P 7d80f46d2eac41e7a9f0b2b2e1972abd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e7aa6d387e94dffaff030eb416c66c4" member_type: VOTER last_known_addr { host: "127.10.198.1" port: 41409 } } peers { permanent_uuid: "fe72d9fd358f4bf7967ac3ea2264aed9" member_type: VOTER last_known_addr { host: "127.10.198.2" port: 41061 } } peers { permanent_uuid: "7d80f46d2eac41e7a9f0b2b2e1972abd" member_type: VOTER last_known_addr { host: "127.10.198.3" port: 37891 } }
I20260925 22:44:59.358170 19070 leader_election.cc:288] T 2069a83262f9446ca244a563aa9ad319 P 7d80f46d2eac41e7a9f0b2b2e1972abd [CANDIDATE]: Term 1 election: Requested vote from peers 4e7aa6d387e94dffaff030eb416c66c4 (127.10.198.1:41409), fe72d9fd358f4bf7967ac3ea2264aed9 (127.10.198.2:41061)
I20260925 22:44:59.358641 18705 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2069a83262f9446ca244a563aa9ad319" candidate_uuid: "7d80f46d2eac41e7a9f0b2b2e1972abd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4e7aa6d387e94dffaff030eb416c66c4"
I20260925 22:44:59.358641 18850 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2069a83262f9446ca244a563aa9ad319" candidate_uuid: "7d80f46d2eac41e7a9f0b2b2e1972abd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe72d9fd358f4bf7967ac3ea2264aed9"
I20260925 22:44:59.358790 18705 raft_consensus.cc:3037] T 2069a83262f9446ca244a563aa9ad319 P 4e7aa6d387e94dffaff030eb416c66c4 [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:44:59.358790 18850 raft_consensus.cc:3037] T 2069a83262f9446ca244a563aa9ad319 P fe72d9fd358f4bf7967ac3ea2264aed9 [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:44:59.360060 18850 raft_consensus.cc:2454] T 2069a83262f9446ca244a563aa9ad319 P fe72d9fd358f4bf7967ac3ea2264aed9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7d80f46d2eac41e7a9f0b2b2e1972abd in term 1.
I20260925 22:44:59.360060 18705 raft_consensus.cc:2454] T 2069a83262f9446ca244a563aa9ad319 P 4e7aa6d387e94dffaff030eb416c66c4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7d80f46d2eac41e7a9f0b2b2e1972abd in term 1.
I20260925 22:44:59.360489 18923 leader_election.cc:302] T 2069a83262f9446ca244a563aa9ad319 P 7d80f46d2eac41e7a9f0b2b2e1972abd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4e7aa6d387e94dffaff030eb416c66c4, 7d80f46d2eac41e7a9f0b2b2e1972abd; no voters:
I20260925 22:44:59.360877 19070 raft_consensus.cc:2781] T 2069a83262f9446ca244a563aa9ad319 P 7d80f46d2eac41e7a9f0b2b2e1972abd [term 1 FOLLOWER]: Leader election won for term 1
I20260925 22:44:59.361110 19070 raft_consensus.cc:686] T 2069a83262f9446ca244a563aa9ad319 P 7d80f46d2eac41e7a9f0b2b2e1972abd [term 1 LEADER]: Becoming Leader. State: Replica: 7d80f46d2eac41e7a9f0b2b2e1972abd, State: Running, Role: LEADER
I20260925 22:44:59.361402 19070 consensus_queue.cc:237] T 2069a83262f9446ca244a563aa9ad319 P 7d80f46d2eac41e7a9f0b2b2e1972abd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e7aa6d387e94dffaff030eb416c66c4" member_type: VOTER last_known_addr { host: "127.10.198.1" port: 41409 } } peers { permanent_uuid: "fe72d9fd358f4bf7967ac3ea2264aed9" member_type: VOTER last_known_addr { host: "127.10.198.2" port: 41061 } } peers { permanent_uuid: "7d80f46d2eac41e7a9f0b2b2e1972abd" member_type: VOTER last_known_addr { host: "127.10.198.3" port: 37891 } }
I20260925 22:44:59.364416 18552 catalog_manager.cc:5495] T 2069a83262f9446ca244a563aa9ad319 P 7d80f46d2eac41e7a9f0b2b2e1972abd reported cstate change: term changed from 0 to 1, leader changed from <none> to 7d80f46d2eac41e7a9f0b2b2e1972abd (127.10.198.3). New cstate: current_term: 1 leader_uuid: "7d80f46d2eac41e7a9f0b2b2e1972abd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e7aa6d387e94dffaff030eb416c66c4" member_type: VOTER last_known_addr { host: "127.10.198.1" port: 41409 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fe72d9fd358f4bf7967ac3ea2264aed9" member_type: VOTER last_known_addr { host: "127.10.198.2" port: 41061 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7d80f46d2eac41e7a9f0b2b2e1972abd" member_type: VOTER last_known_addr { host: "127.10.198.3" port: 37891 } health_report { overall_health: HEALTHY } } }
I20260925 22:44:59.392663 19144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:44:59.388048 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60542 (local address 127.10.198.3:37891)
0925 22:44:59.388376 (+ 328us) server_negotiation.cc:206] Beginning negotiation
0925 22:44:59.388383 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 22:44:59.388398 (+ 15us) server_negotiation.cc:407] Connection header received
0925 22:44:59.388484 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:44:59.388486 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:44:59.388545 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:44:59.388731 (+ 186us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 22:44:59.388962 (+ 231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:44:59.389558 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:44:59.390649 (+ 1091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:44:59.390993 (+ 344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:44:59.391088 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:44:59.391413 (+ 325us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 22:44:59.392097 (+ 684us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 22:44:59.392223 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0925 22:44:59.392357 (+ 134us) server_negotiation.cc:299] Negotiation successful
0925 22:44:59.392446 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":246,"thread_start_us":157,"threads_started":1}
I20260925 22:44:59.395803 18705 raft_consensus.cc:1261] T 2069a83262f9446ca244a563aa9ad319 P 4e7aa6d387e94dffaff030eb416c66c4 [term 1 FOLLOWER]: Refusing update from remote peer 7d80f46d2eac41e7a9f0b2b2e1972abd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 22:44:59.396008 18850 raft_consensus.cc:1261] T 2069a83262f9446ca244a563aa9ad319 P fe72d9fd358f4bf7967ac3ea2264aed9 [term 1 FOLLOWER]: Refusing update from remote peer 7d80f46d2eac41e7a9f0b2b2e1972abd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 22:44:59.396425 19070 consensus_queue.cc:1035] T 2069a83262f9446ca244a563aa9ad319 P 7d80f46d2eac41e7a9f0b2b2e1972abd [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e7aa6d387e94dffaff030eb416c66c4" member_type: VOTER last_known_addr { host: "127.10.198.1" port: 41409 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 22:44:59.396665 19141 consensus_queue.cc:1035] T 2069a83262f9446ca244a563aa9ad319 P 7d80f46d2eac41e7a9f0b2b2e1972abd [LEADER]: Connected to new peer: Peer: permanent_uuid: "fe72d9fd358f4bf7967ac3ea2264aed9" member_type: VOTER last_known_addr { host: "127.10.198.2" port: 41061 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 22:44:59.411790 18552 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55556:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 22:44:59.412045 18552 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55556:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 22:44:59.414582 18552 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9a42cb7f7adf47f1b0dd00bf3bfbd491: Sending DeleteTablet for 3 replicas of tablet 2069a83262f9446ca244a563aa9ad319
I20260925 22:44:59.415329 18683 tablet_service.cc:1514] Processing DeleteTablet for tablet 2069a83262f9446ca244a563aa9ad319 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 22:44:59 UTC) from {username='oryx', principal='oryx/127.10.198.62@KRBTEST.COM'} at 127.0.0.1:39244
I20260925 22:44:59.415632 19088 tablet_replica.cc:330] T 2069a83262f9446ca244a563aa9ad319 P 4e7aa6d387e94dffaff030eb416c66c4: stopping tablet replica
I20260925 22:44:59.415599 18830 tablet_service.cc:1514] Processing DeleteTablet for tablet 2069a83262f9446ca244a563aa9ad319 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 22:44:59 UTC) from {username='oryx', principal='oryx/127.10.198.62@KRBTEST.COM'} at 127.0.0.1:49946
I20260925 22:44:59.415807 19089 tablet_replica.cc:330] T 2069a83262f9446ca244a563aa9ad319 P fe72d9fd358f4bf7967ac3ea2264aed9: stopping tablet replica
I20260925 22:44:59.415828 19088 raft_consensus.cc:2229] T 2069a83262f9446ca244a563aa9ad319 P 4e7aa6d387e94dffaff030eb416c66c4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 22:44:59.415992 19089 raft_consensus.cc:2229] T 2069a83262f9446ca244a563aa9ad319 P fe72d9fd358f4bf7967ac3ea2264aed9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 22:44:59.416087 19088 raft_consensus.cc:2258] T 2069a83262f9446ca244a563aa9ad319 P 4e7aa6d387e94dffaff030eb416c66c4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 22:44:59.416178 19089 raft_consensus.cc:2258] T 2069a83262f9446ca244a563aa9ad319 P fe72d9fd358f4bf7967ac3ea2264aed9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 22:44:59.416621 18970 tablet_service.cc:1514] Processing DeleteTablet for tablet 2069a83262f9446ca244a563aa9ad319 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 22:44:59 UTC) from {username='oryx', principal='oryx/127.10.198.62@KRBTEST.COM'} at 127.0.0.1:60530
I20260925 22:44:59.416891 19090 tablet_replica.cc:330] T 2069a83262f9446ca244a563aa9ad319 P 7d80f46d2eac41e7a9f0b2b2e1972abd: stopping tablet replica
I20260925 22:44:59.417006 19088 ts_tablet_manager.cc:1905] T 2069a83262f9446ca244a563aa9ad319 P 4e7aa6d387e94dffaff030eb416c66c4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 22:44:59.417039 19090 raft_consensus.cc:2229] T 2069a83262f9446ca244a563aa9ad319 P 7d80f46d2eac41e7a9f0b2b2e1972abd [term 1 LEADER]: Raft consensus shutting down.
I20260925 22:44:59.417394 19090 raft_consensus.cc:2258] T 2069a83262f9446ca244a563aa9ad319 P 7d80f46d2eac41e7a9f0b2b2e1972abd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 22:44:59.418102 19090 ts_tablet_manager.cc:1905] T 2069a83262f9446ca244a563aa9ad319 P 7d80f46d2eac41e7a9f0b2b2e1972abd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 22:44:59.418529 19089 ts_tablet_manager.cc:1905] T 2069a83262f9446ca244a563aa9ad319 P fe72d9fd358f4bf7967ac3ea2264aed9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 22:44:59.419982 19088 ts_tablet_manager.cc:1918] T 2069a83262f9446ca244a563aa9ad319 P 4e7aa6d387e94dffaff030eb416c66c4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 22:44:59.420097 19088 log.cc:1196] T 2069a83262f9446ca244a563aa9ad319 P 4e7aa6d387e94dffaff030eb416c66c4: Deleting WAL directory at /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790376259650642-11032-0/minicluster-data/ts-0/wal/wals/2069a83262f9446ca244a563aa9ad319
I20260925 22:44:59.420416 19088 ts_tablet_manager.cc:1939] T 2069a83262f9446ca244a563aa9ad319 P 4e7aa6d387e94dffaff030eb416c66c4: Deleting consensus metadata
I20260925 22:44:59.421373 19090 ts_tablet_manager.cc:1918] T 2069a83262f9446ca244a563aa9ad319 P 7d80f46d2eac41e7a9f0b2b2e1972abd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 22:44:59.421363 18539 catalog_manager.cc:4841] TS 4e7aa6d387e94dffaff030eb416c66c4 (127.10.198.1:41409): tablet 2069a83262f9446ca244a563aa9ad319 (table test-table [id=899ab479aa0e4676b6bd01c1931e0f07]) successfully deleted
I20260925 22:44:59.421446 19090 log.cc:1196] T 2069a83262f9446ca244a563aa9ad319 P 7d80f46d2eac41e7a9f0b2b2e1972abd: Deleting WAL directory at /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790376259650642-11032-0/minicluster-data/ts-2/wal/wals/2069a83262f9446ca244a563aa9ad319
I20260925 22:44:59.421713 19090 ts_tablet_manager.cc:1939] T 2069a83262f9446ca244a563aa9ad319 P 7d80f46d2eac41e7a9f0b2b2e1972abd: Deleting consensus metadata
I20260925 22:44:59.421736 19089 ts_tablet_manager.cc:1918] T 2069a83262f9446ca244a563aa9ad319 P fe72d9fd358f4bf7967ac3ea2264aed9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 22:44:59.421811 19089 log.cc:1196] T 2069a83262f9446ca244a563aa9ad319 P fe72d9fd358f4bf7967ac3ea2264aed9: Deleting WAL directory at /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790376259650642-11032-0/minicluster-data/ts-1/wal/wals/2069a83262f9446ca244a563aa9ad319
I20260925 22:44:59.422044 19089 ts_tablet_manager.cc:1939] T 2069a83262f9446ca244a563aa9ad319 P fe72d9fd358f4bf7967ac3ea2264aed9: Deleting consensus metadata
I20260925 22:44:59.421931 18552 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55556:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 22:44:59.422436 18552 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 22:44:59.422466 18540 catalog_manager.cc:4841] TS 7d80f46d2eac41e7a9f0b2b2e1972abd (127.10.198.3:37891): tablet 2069a83262f9446ca244a563aa9ad319 (table test-table [id=899ab479aa0e4676b6bd01c1931e0f07]) successfully deleted
I20260925 22:44:59.422724 18540 catalog_manager.cc:4841] TS fe72d9fd358f4bf7967ac3ea2264aed9 (127.10.198.2:41061): tablet 2069a83262f9446ca244a563aa9ad319 (table test-table [id=899ab479aa0e4676b6bd01c1931e0f07]) successfully deleted
I20260925 22:44:59.427906 18683 tablet_service.cc:1467] Processing CreateTablet for tablet 2db39ca87d1a4e35bc998faa7d7790d9 (DEFAULT_TABLE table=test-table [id=e272cc7f71564432a1450f7f4147ee3f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 22:44:59.428006 18830 tablet_service.cc:1467] Processing CreateTablet for tablet 2db39ca87d1a4e35bc998faa7d7790d9 (DEFAULT_TABLE table=test-table [id=e272cc7f71564432a1450f7f4147ee3f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 22:44:59.428277 18683 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2db39ca87d1a4e35bc998faa7d7790d9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:44:59.428220 18970 tablet_service.cc:1467] Processing CreateTablet for tablet 2db39ca87d1a4e35bc998faa7d7790d9 (DEFAULT_TABLE table=test-table [id=e272cc7f71564432a1450f7f4147ee3f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 22:44:59.428340 18830 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2db39ca87d1a4e35bc998faa7d7790d9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:44:59.428464 18970 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2db39ca87d1a4e35bc998faa7d7790d9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:44:59.431093 19064 tablet_bootstrap.cc:492] T 2db39ca87d1a4e35bc998faa7d7790d9 P 7d80f46d2eac41e7a9f0b2b2e1972abd: Bootstrap starting.
I20260925 22:44:59.431106 19065 tablet_bootstrap.cc:492] T 2db39ca87d1a4e35bc998faa7d7790d9 P 4e7aa6d387e94dffaff030eb416c66c4: Bootstrap starting.
I20260925 22:44:59.432179 19066 tablet_bootstrap.cc:492] T 2db39ca87d1a4e35bc998faa7d7790d9 P fe72d9fd358f4bf7967ac3ea2264aed9: Bootstrap starting.
I20260925 22:44:59.432228 19065 tablet_bootstrap.cc:654] T 2db39ca87d1a4e35bc998faa7d7790d9 P 4e7aa6d387e94dffaff030eb416c66c4: Neither blocks nor log segments found. Creating new log.
I20260925 22:44:59.432229 19064 tablet_bootstrap.cc:654] T 2db39ca87d1a4e35bc998faa7d7790d9 P 7d80f46d2eac41e7a9f0b2b2e1972abd: Neither blocks nor log segments found. Creating new log.
I20260925 22:44:59.433153 19066 tablet_bootstrap.cc:654] T 2db39ca87d1a4e35bc998faa7d7790d9 P fe72d9fd358f4bf7967ac3ea2264aed9: Neither blocks nor log segments found. Creating new log.
I20260925 22:44:59.433519 19064 tablet_bootstrap.cc:492] T 2db39ca87d1a4e35bc998faa7d7790d9 P 7d80f46d2eac41e7a9f0b2b2e1972abd: No bootstrap required, opened a new log
I20260925 22:44:59.433650 19064 ts_tablet_manager.cc:1397] T 2db39ca87d1a4e35bc998faa7d7790d9 P 7d80f46d2eac41e7a9f0b2b2e1972abd: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260925 22:44:59.434151 19064 raft_consensus.cc:348] T 2db39ca87d1a4e35bc998faa7d7790d9 P 7d80f46d2eac41e7a9f0b2b2e1972abd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e7aa6d387e94dffaff030eb416c66c4" member_type: VOTER last_known_addr { host: "127.10.198.1" port: 41409 } } peers { permanent_uuid: "fe72d9fd358f4bf7967ac3ea2264aed9" member_type: VOTER last_known_addr { host: "127.10.198.2" port: 41061 } } peers { permanent_uuid: "7d80f46d2eac41e7a9f0b2b2e1972abd" member_type: VOTER last_known_addr { host: "127.10.198.3" port: 37891 } }
I20260925 22:44:59.434290 19064 raft_consensus.cc:374] T 2db39ca87d1a4e35bc998faa7d7790d9 P 7d80f46d2eac41e7a9f0b2b2e1972abd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:44:59.434335 19064 raft_consensus.cc:729] T 2db39ca87d1a4e35bc998faa7d7790d9 P 7d80f46d2eac41e7a9f0b2b2e1972abd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7d80f46d2eac41e7a9f0b2b2e1972abd, State: Initialized, Role: FOLLOWER
I20260925 22:44:59.434479 19064 consensus_queue.cc:260] T 2db39ca87d1a4e35bc998faa7d7790d9 P 7d80f46d2eac41e7a9f0b2b2e1972abd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e7aa6d387e94dffaff030eb416c66c4" member_type: VOTER last_known_addr { host: "127.10.198.1" port: 41409 } } peers { permanent_uuid: "fe72d9fd358f4bf7967ac3ea2264aed9" member_type: VOTER last_known_addr { host: "127.10.198.2" port: 41061 } } peers { permanent_uuid: "7d80f46d2eac41e7a9f0b2b2e1972abd" member_type: VOTER last_known_addr { host: "127.10.198.3" port: 37891 } }
I20260925 22:44:59.434804 19064 ts_tablet_manager.cc:1428] T 2db39ca87d1a4e35bc998faa7d7790d9 P 7d80f46d2eac41e7a9f0b2b2e1972abd: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260925 22:44:59.435428 19065 tablet_bootstrap.cc:492] T 2db39ca87d1a4e35bc998faa7d7790d9 P 4e7aa6d387e94dffaff030eb416c66c4: No bootstrap required, opened a new log
I20260925 22:44:59.435514 19065 ts_tablet_manager.cc:1397] T 2db39ca87d1a4e35bc998faa7d7790d9 P 4e7aa6d387e94dffaff030eb416c66c4: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.001s
I20260925 22:44:59.436218 19065 raft_consensus.cc:348] T 2db39ca87d1a4e35bc998faa7d7790d9 P 4e7aa6d387e94dffaff030eb416c66c4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e7aa6d387e94dffaff030eb416c66c4" member_type: VOTER last_known_addr { host: "127.10.198.1" port: 41409 } } peers { permanent_uuid: "fe72d9fd358f4bf7967ac3ea2264aed9" member_type: VOTER last_known_addr { host: "127.10.198.2" port: 41061 } } peers { permanent_uuid: "7d80f46d2eac41e7a9f0b2b2e1972abd" member_type: VOTER last_known_addr { host: "127.10.198.3" port: 37891 } }
I20260925 22:44:59.436398 19065 raft_consensus.cc:374] T 2db39ca87d1a4e35bc998faa7d7790d9 P 4e7aa6d387e94dffaff030eb416c66c4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:44:59.436452 19065 raft_consensus.cc:729] T 2db39ca87d1a4e35bc998faa7d7790d9 P 4e7aa6d387e94dffaff030eb416c66c4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4e7aa6d387e94dffaff030eb416c66c4, State: Initialized, Role: FOLLOWER
I20260925 22:44:59.436604 19065 consensus_queue.cc:260] T 2db39ca87d1a4e35bc998faa7d7790d9 P 4e7aa6d387e94dffaff030eb416c66c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e7aa6d387e94dffaff030eb416c66c4" member_type: VOTER last_known_addr { host: "127.10.198.1" port: 41409 } } peers { permanent_uuid: "fe72d9fd358f4bf7967ac3ea2264aed9" member_type: VOTER last_known_addr { host: "127.10.198.2" port: 41061 } } peers { permanent_uuid: "7d80f46d2eac41e7a9f0b2b2e1972abd" member_type: VOTER last_known_addr { host: "127.10.198.3" port: 37891 } }
I20260925 22:44:59.436904 19065 ts_tablet_manager.cc:1428] T 2db39ca87d1a4e35bc998faa7d7790d9 P 4e7aa6d387e94dffaff030eb416c66c4: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260925 22:44:59.440984 19066 tablet_bootstrap.cc:492] T 2db39ca87d1a4e35bc998faa7d7790d9 P fe72d9fd358f4bf7967ac3ea2264aed9: No bootstrap required, opened a new log
I20260925 22:44:59.441128 19066 ts_tablet_manager.cc:1397] T 2db39ca87d1a4e35bc998faa7d7790d9 P fe72d9fd358f4bf7967ac3ea2264aed9: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.000s
I20260925 22:44:59.441646 19066 raft_consensus.cc:348] T 2db39ca87d1a4e35bc998faa7d7790d9 P fe72d9fd358f4bf7967ac3ea2264aed9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e7aa6d387e94dffaff030eb416c66c4" member_type: VOTER last_known_addr { host: "127.10.198.1" port: 41409 } } peers { permanent_uuid: "fe72d9fd358f4bf7967ac3ea2264aed9" member_type: VOTER last_known_addr { host: "127.10.198.2" port: 41061 } } peers { permanent_uuid: "7d80f46d2eac41e7a9f0b2b2e1972abd" member_type: VOTER last_known_addr { host: "127.10.198.3" port: 37891 } }
I20260925 22:44:59.441799 19066 raft_consensus.cc:374] T 2db39ca87d1a4e35bc998faa7d7790d9 P fe72d9fd358f4bf7967ac3ea2264aed9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:44:59.441836 19066 raft_consensus.cc:729] T 2db39ca87d1a4e35bc998faa7d7790d9 P fe72d9fd358f4bf7967ac3ea2264aed9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe72d9fd358f4bf7967ac3ea2264aed9, State: Initialized, Role: FOLLOWER
I20260925 22:44:59.441995 19066 consensus_queue.cc:260] T 2db39ca87d1a4e35bc998faa7d7790d9 P fe72d9fd358f4bf7967ac3ea2264aed9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e7aa6d387e94dffaff030eb416c66c4" member_type: VOTER last_known_addr { host: "127.10.198.1" port: 41409 } } peers { permanent_uuid: "fe72d9fd358f4bf7967ac3ea2264aed9" member_type: VOTER last_known_addr { host: "127.10.198.2" port: 41061 } } peers { permanent_uuid: "7d80f46d2eac41e7a9f0b2b2e1972abd" member_type: VOTER last_known_addr { host: "127.10.198.3" port: 37891 } }
I20260925 22:44:59.442428 19066 ts_tablet_manager.cc:1428] T 2db39ca87d1a4e35bc998faa7d7790d9 P fe72d9fd358f4bf7967ac3ea2264aed9: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20260925 22:44:59.568787 19072 raft_consensus.cc:482] T 2db39ca87d1a4e35bc998faa7d7790d9 P fe72d9fd358f4bf7967ac3ea2264aed9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 22:44:59.568972 19072 raft_consensus.cc:504] T 2db39ca87d1a4e35bc998faa7d7790d9 P fe72d9fd358f4bf7967ac3ea2264aed9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e7aa6d387e94dffaff030eb416c66c4" member_type: VOTER last_known_addr { host: "127.10.198.1" port: 41409 } } peers { permanent_uuid: "fe72d9fd358f4bf7967ac3ea2264aed9" member_type: VOTER last_known_addr { host: "127.10.198.2" port: 41061 } } peers { permanent_uuid: "7d80f46d2eac41e7a9f0b2b2e1972abd" member_type: VOTER last_known_addr { host: "127.10.198.3" port: 37891 } }
I20260925 22:44:59.569481 19072 leader_election.cc:288] T 2db39ca87d1a4e35bc998faa7d7790d9 P fe72d9fd358f4bf7967ac3ea2264aed9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4e7aa6d387e94dffaff030eb416c66c4 (127.10.198.1:41409), 7d80f46d2eac41e7a9f0b2b2e1972abd (127.10.198.3:37891)
I20260925 22:44:59.569938 18705 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2db39ca87d1a4e35bc998faa7d7790d9" candidate_uuid: "fe72d9fd358f4bf7967ac3ea2264aed9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4e7aa6d387e94dffaff030eb416c66c4" is_pre_election: true
I20260925 22:44:59.570030 18989 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2db39ca87d1a4e35bc998faa7d7790d9" candidate_uuid: "fe72d9fd358f4bf7967ac3ea2264aed9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7d80f46d2eac41e7a9f0b2b2e1972abd" is_pre_election: true
I20260925 22:44:59.570109 18705 raft_consensus.cc:2454] T 2db39ca87d1a4e35bc998faa7d7790d9 P 4e7aa6d387e94dffaff030eb416c66c4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fe72d9fd358f4bf7967ac3ea2264aed9 in term 0.
I20260925 22:44:59.570153 18989 raft_consensus.cc:2454] T 2db39ca87d1a4e35bc998faa7d7790d9 P 7d80f46d2eac41e7a9f0b2b2e1972abd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fe72d9fd358f4bf7967ac3ea2264aed9 in term 0.
I20260925 22:44:59.570468 18783 leader_election.cc:302] T 2db39ca87d1a4e35bc998faa7d7790d9 P fe72d9fd358f4bf7967ac3ea2264aed9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4e7aa6d387e94dffaff030eb416c66c4, fe72d9fd358f4bf7967ac3ea2264aed9; no voters:
I20260925 22:44:59.570669 19072 raft_consensus.cc:2781] T 2db39ca87d1a4e35bc998faa7d7790d9 P fe72d9fd358f4bf7967ac3ea2264aed9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 22:44:59.570730 19072 raft_consensus.cc:482] T 2db39ca87d1a4e35bc998faa7d7790d9 P fe72d9fd358f4bf7967ac3ea2264aed9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 22:44:59.570784 19072 raft_consensus.cc:3037] T 2db39ca87d1a4e35bc998faa7d7790d9 P fe72d9fd358f4bf7967ac3ea2264aed9 [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:44:59.571930 19072 raft_consensus.cc:504] T 2db39ca87d1a4e35bc998faa7d7790d9 P fe72d9fd358f4bf7967ac3ea2264aed9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e7aa6d387e94dffaff030eb416c66c4" member_type: VOTER last_known_addr { host: "127.10.198.1" port: 41409 } } peers { permanent_uuid: "fe72d9fd358f4bf7967ac3ea2264aed9" member_type: VOTER last_known_addr { host: "127.10.198.2" port: 41061 } } peers { permanent_uuid: "7d80f46d2eac41e7a9f0b2b2e1972abd" member_type: VOTER last_known_addr { host: "127.10.198.3" port: 37891 } }
I20260925 22:44:59.572399 19072 leader_election.cc:288] T 2db39ca87d1a4e35bc998faa7d7790d9 P fe72d9fd358f4bf7967ac3ea2264aed9 [CANDIDATE]: Term 1 election: Requested vote from peers 4e7aa6d387e94dffaff030eb416c66c4 (127.10.198.1:41409), 7d80f46d2eac41e7a9f0b2b2e1972abd (127.10.198.3:37891)
I20260925 22:44:59.572788 18989 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2db39ca87d1a4e35bc998faa7d7790d9" candidate_uuid: "fe72d9fd358f4bf7967ac3ea2264aed9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7d80f46d2eac41e7a9f0b2b2e1972abd"
I20260925 22:44:59.572803 18705 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2db39ca87d1a4e35bc998faa7d7790d9" candidate_uuid: "fe72d9fd358f4bf7967ac3ea2264aed9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4e7aa6d387e94dffaff030eb416c66c4"
I20260925 22:44:59.572935 18989 raft_consensus.cc:3037] T 2db39ca87d1a4e35bc998faa7d7790d9 P 7d80f46d2eac41e7a9f0b2b2e1972abd [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:44:59.572978 18705 raft_consensus.cc:3037] T 2db39ca87d1a4e35bc998faa7d7790d9 P 4e7aa6d387e94dffaff030eb416c66c4 [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:44:59.574023 18705 raft_consensus.cc:2454] T 2db39ca87d1a4e35bc998faa7d7790d9 P 4e7aa6d387e94dffaff030eb416c66c4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fe72d9fd358f4bf7967ac3ea2264aed9 in term 1.
I20260925 22:44:59.574194 18989 raft_consensus.cc:2454] T 2db39ca87d1a4e35bc998faa7d7790d9 P 7d80f46d2eac41e7a9f0b2b2e1972abd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fe72d9fd358f4bf7967ac3ea2264aed9 in term 1.
I20260925 22:44:59.574455 18783 leader_election.cc:302] T 2db39ca87d1a4e35bc998faa7d7790d9 P fe72d9fd358f4bf7967ac3ea2264aed9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4e7aa6d387e94dffaff030eb416c66c4, fe72d9fd358f4bf7967ac3ea2264aed9; no voters:
I20260925 22:44:59.574707 19072 raft_consensus.cc:2781] T 2db39ca87d1a4e35bc998faa7d7790d9 P fe72d9fd358f4bf7967ac3ea2264aed9 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 22:44:59.574815 19072 raft_consensus.cc:686] T 2db39ca87d1a4e35bc998faa7d7790d9 P fe72d9fd358f4bf7967ac3ea2264aed9 [term 1 LEADER]: Becoming Leader. State: Replica: fe72d9fd358f4bf7967ac3ea2264aed9, State: Running, Role: LEADER
I20260925 22:44:59.574968 19072 consensus_queue.cc:237] T 2db39ca87d1a4e35bc998faa7d7790d9 P fe72d9fd358f4bf7967ac3ea2264aed9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e7aa6d387e94dffaff030eb416c66c4" member_type: VOTER last_known_addr { host: "127.10.198.1" port: 41409 } } peers { permanent_uuid: "fe72d9fd358f4bf7967ac3ea2264aed9" member_type: VOTER last_known_addr { host: "127.10.198.2" port: 41061 } } peers { permanent_uuid: "7d80f46d2eac41e7a9f0b2b2e1972abd" member_type: VOTER last_known_addr { host: "127.10.198.3" port: 37891 } }
I20260925 22:44:59.576843 18551 catalog_manager.cc:5495] T 2db39ca87d1a4e35bc998faa7d7790d9 P fe72d9fd358f4bf7967ac3ea2264aed9 reported cstate change: term changed from 0 to 1, leader changed from <none> to fe72d9fd358f4bf7967ac3ea2264aed9 (127.10.198.2). New cstate: current_term: 1 leader_uuid: "fe72d9fd358f4bf7967ac3ea2264aed9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e7aa6d387e94dffaff030eb416c66c4" member_type: VOTER last_known_addr { host: "127.10.198.1" port: 41409 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fe72d9fd358f4bf7967ac3ea2264aed9" member_type: VOTER last_known_addr { host: "127.10.198.2" port: 41061 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7d80f46d2eac41e7a9f0b2b2e1972abd" member_type: VOTER last_known_addr { host: "127.10.198.3" port: 37891 } health_report { overall_health: UNKNOWN } } }
I20260925 22:44:59.605696 19062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:44:59.602646 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49972 (local address 127.10.198.2:41061)
0925 22:44:59.602771 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0925 22:44:59.602774 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 22:44:59.602788 (+ 14us) server_negotiation.cc:407] Connection header received
0925 22:44:59.602834 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:44:59.602837 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:44:59.602880 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:44:59.602950 (+ 70us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 22:44:59.603211 (+ 261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:44:59.603721 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:44:59.604646 (+ 925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:44:59.604883 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:44:59.604977 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:44:59.605225 (+ 248us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 22:44:59.605298 (+ 73us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 22:44:59.605355 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0925 22:44:59.605520 (+ 165us) server_negotiation.cc:299] Negotiation successful
0925 22:44:59.605573 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260925 22:44:59.615422 19121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:44:59.611955 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.198.2:41061 (local address 127.10.198.2:35289)
0925 22:44:59.612098 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0925 22:44:59.612114 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 22:44:59.612234 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:44:59.612466 (+ 232us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:44:59.612468 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:44:59.612475 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 22:44:59.612622 (+ 147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:44:59.612627 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:44:59.613408 (+ 781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:44:59.613414 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:44:59.614149 (+ 735us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:44:59.614160 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:44:59.614619 (+ 459us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:44:59.614622 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:44:59.614699 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:44:59.614732 (+ 33us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 22:44:59.615052 (+ 320us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 22:44:59.615059 (+ 7us) client_negotiation.cc:769] Sending connection context
0925 22:44:59.615120 (+ 61us) client_negotiation.cc:240] Negotiation successful
0925 22:44:59.615264 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":44,"mutex_wait_us":48}
I20260925 22:44:59.615422 19062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:44:59.612146 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.198.2:35289 (local address 127.10.198.2:41061)
0925 22:44:59.612255 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0925 22:44:59.612260 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:44:59.612279 (+ 19us) server_negotiation.cc:407] Connection header received
0925 22:44:59.612321 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:44:59.612324 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:44:59.612369 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:44:59.612450 (+ 81us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 22:44:59.612727 (+ 277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:44:59.613282 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:44:59.614279 (+ 997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:44:59.614498 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:44:59.614613 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:44:59.614881 (+ 268us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 22:44:59.614959 (+ 78us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 22:44:59.615039 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0925 22:44:59.615127 (+ 88us) server_negotiation.cc:299] Negotiation successful
0925 22:44:59.615249 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":49,"server-negotiator.queue_time_us":49}
I20260925 22:44:59.621939 18989 raft_consensus.cc:1261] T 2db39ca87d1a4e35bc998faa7d7790d9 P 7d80f46d2eac41e7a9f0b2b2e1972abd [term 1 FOLLOWER]: Refusing update from remote peer fe72d9fd358f4bf7967ac3ea2264aed9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 22:44:59.622035 18705 raft_consensus.cc:1261] T 2db39ca87d1a4e35bc998faa7d7790d9 P 4e7aa6d387e94dffaff030eb416c66c4 [term 1 FOLLOWER]: Refusing update from remote peer fe72d9fd358f4bf7967ac3ea2264aed9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 22:44:59.622367 19072 consensus_queue.cc:1035] T 2db39ca87d1a4e35bc998faa7d7790d9 P fe72d9fd358f4bf7967ac3ea2264aed9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7d80f46d2eac41e7a9f0b2b2e1972abd" member_type: VOTER last_known_addr { host: "127.10.198.3" port: 37891 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 22:44:59.622525 19083 consensus_queue.cc:1035] T 2db39ca87d1a4e35bc998faa7d7790d9 P fe72d9fd358f4bf7967ac3ea2264aed9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e7aa6d387e94dffaff030eb416c66c4" member_type: VOTER last_known_addr { host: "127.10.198.1" port: 41409 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 22:44:59.657593 18553 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55556:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 22:44:59.657761 18553 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55556:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 22:44:59.659830 18553 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9a42cb7f7adf47f1b0dd00bf3bfbd491: Sending DeleteTablet for 3 replicas of tablet 2db39ca87d1a4e35bc998faa7d7790d9
I20260925 22:44:59.660475 18683 tablet_service.cc:1514] Processing DeleteTablet for tablet 2db39ca87d1a4e35bc998faa7d7790d9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 22:44:59 UTC) from {username='oryx', principal='oryx/127.10.198.62@KRBTEST.COM'} at 127.0.0.1:39244
I20260925 22:44:59.660595 18970 tablet_service.cc:1514] Processing DeleteTablet for tablet 2db39ca87d1a4e35bc998faa7d7790d9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 22:44:59 UTC) from {username='oryx', principal='oryx/127.10.198.62@KRBTEST.COM'} at 127.0.0.1:60530
I20260925 22:44:59.660692 19088 tablet_replica.cc:330] T 2db39ca87d1a4e35bc998faa7d7790d9 P 4e7aa6d387e94dffaff030eb416c66c4: stopping tablet replica
I20260925 22:44:59.660827 19088 raft_consensus.cc:2229] T 2db39ca87d1a4e35bc998faa7d7790d9 P 4e7aa6d387e94dffaff030eb416c66c4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 22:44:59.660918 19090 tablet_replica.cc:330] T 2db39ca87d1a4e35bc998faa7d7790d9 P 7d80f46d2eac41e7a9f0b2b2e1972abd: stopping tablet replica
I20260925 22:44:59.660972 19088 raft_consensus.cc:2258] T 2db39ca87d1a4e35bc998faa7d7790d9 P 4e7aa6d387e94dffaff030eb416c66c4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 22:44:59.661048 19090 raft_consensus.cc:2229] T 2db39ca87d1a4e35bc998faa7d7790d9 P 7d80f46d2eac41e7a9f0b2b2e1972abd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 22:44:59.661142 11032 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrWXm8X/build/debug/bin/kudu with pid 18625
I20260925 22:44:59.661188 19090 raft_consensus.cc:2258] T 2db39ca87d1a4e35bc998faa7d7790d9 P 7d80f46d2eac41e7a9f0b2b2e1972abd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 22:44:59.662317 18829 tablet_service.cc:1514] Processing DeleteTablet for tablet 2db39ca87d1a4e35bc998faa7d7790d9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 22:44:59 UTC) from {username='oryx', principal='oryx/127.10.198.62@KRBTEST.COM'} at 127.0.0.1:49946
I20260925 22:44:59.662532 19089 tablet_replica.cc:330] T 2db39ca87d1a4e35bc998faa7d7790d9 P fe72d9fd358f4bf7967ac3ea2264aed9: stopping tablet replica
I20260925 22:44:59.662693 19089 raft_consensus.cc:2229] T 2db39ca87d1a4e35bc998faa7d7790d9 P fe72d9fd358f4bf7967ac3ea2264aed9 [term 1 LEADER]: Raft consensus shutting down.
I20260925 22:44:59.662945 19089 raft_consensus.cc:2258] T 2db39ca87d1a4e35bc998faa7d7790d9 P fe72d9fd358f4bf7967ac3ea2264aed9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 22:44:59.663738 19089 ts_tablet_manager.cc:1905] T 2db39ca87d1a4e35bc998faa7d7790d9 P fe72d9fd358f4bf7967ac3ea2264aed9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 22:44:59.666471 19089 ts_tablet_manager.cc:1918] T 2db39ca87d1a4e35bc998faa7d7790d9 P fe72d9fd358f4bf7967ac3ea2264aed9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 22:44:59.666550 19089 log.cc:1196] T 2db39ca87d1a4e35bc998faa7d7790d9 P fe72d9fd358f4bf7967ac3ea2264aed9: Deleting WAL directory at /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790376259650642-11032-0/minicluster-data/ts-1/wal/wals/2db39ca87d1a4e35bc998faa7d7790d9
I20260925 22:44:59.666806 19089 ts_tablet_manager.cc:1939] T 2db39ca87d1a4e35bc998faa7d7790d9 P fe72d9fd358f4bf7967ac3ea2264aed9: Deleting consensus metadata
I20260925 22:44:59.667627 18540 catalog_manager.cc:4841] TS fe72d9fd358f4bf7967ac3ea2264aed9 (127.10.198.2:41061): tablet 2db39ca87d1a4e35bc998faa7d7790d9 (table test-table [id=e272cc7f71564432a1450f7f4147ee3f]) successfully deleted
I20260925 22:44:59.668459 19090 ts_tablet_manager.cc:1905] T 2db39ca87d1a4e35bc998faa7d7790d9 P 7d80f46d2eac41e7a9f0b2b2e1972abd: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260925 22:44:59.670574 18539 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.10.198.1:41409 (error 108)
I20260925 22:44:59.670950 19090 ts_tablet_manager.cc:1918] T 2db39ca87d1a4e35bc998faa7d7790d9 P 7d80f46d2eac41e7a9f0b2b2e1972abd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 22:44:59.671041 19090 log.cc:1196] T 2db39ca87d1a4e35bc998faa7d7790d9 P 7d80f46d2eac41e7a9f0b2b2e1972abd: Deleting WAL directory at /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790376259650642-11032-0/minicluster-data/ts-2/wal/wals/2db39ca87d1a4e35bc998faa7d7790d9
I20260925 22:44:59.671252 11032 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrWXm8X/build/debug/bin/kudu with pid 18765
I20260925 22:44:59.671319 19090 ts_tablet_manager.cc:1939] T 2db39ca87d1a4e35bc998faa7d7790d9 P 7d80f46d2eac41e7a9f0b2b2e1972abd: Deleting consensus metadata
I20260925 22:44:59.673439 18540 catalog_manager.cc:4841] TS 7d80f46d2eac41e7a9f0b2b2e1972abd (127.10.198.3:37891): tablet 2db39ca87d1a4e35bc998faa7d7790d9 (table test-table [id=e272cc7f71564432a1450f7f4147ee3f]) successfully deleted
I20260925 22:44:59.674371 19159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:44:59.673952 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.198.1:41409 (local address 127.0.0.1:39270)
0925 22:44:59.674190 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0925 22:44:59.674283 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.198.1:41409: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":88,"threads_started":1}
W20260925 22:44:59.674526 18539 catalog_manager.cc:4584] TS 4e7aa6d387e94dffaff030eb416c66c4 (127.10.198.1:41409): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 2db39ca87d1a4e35bc998faa7d7790d9: Network error: Client connection negotiation failed: client connection to 127.10.198.1:41409: connect: Connection refused (error 111)
I20260925 22:44:59.679329 11032 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrWXm8X/build/debug/bin/kudu with pid 18905
I20260925 22:44:59.687096 11032 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrWXm8X/build/debug/bin/kudu with pid 18521
2026-09-25T22:44:59Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (3584 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790376259650642-11032-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 22:44:59 dist-test-slave-5vdr krb5kdc[19162](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 22:44:59 dist-test-slave-5vdr krb5kdc[19162](info): set up 2 sockets
Sep 25 22:44:59 dist-test-slave-5vdr krb5kdc[19162](info): commencing operation
krb5kdc: starting...
W20260925 22:45:00.752722 11032 mini_kdc.cc:121] Time spent starting KDC: real 1.042s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790376259650642-11032-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790376259650642-11032-0/krb5kdc/test-user.keytab.
Sep 25 22:45:00 dist-test-slave-5vdr krb5kdc[19162](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376300, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T22:45:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T22:45:00Z Disabled control of system clock
WARNING: no policy specified for kudu/127.10.198.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.198.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.198.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790376259650642-11032-0/krb5kdc/kudu_127.10.198.62.keytab.
Entry for principal kudu/127.10.198.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790376259650642-11032-0/krb5kdc/kudu_127.10.198.62.keytab.
I20260925 22:45:00.889426 11032 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrWXm8X/build/debug/bin/kudu
/tmp/dist-test-taskrWXm8X/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790376259650642-11032-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790376259650642-11032-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790376259650642-11032-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790376259650642-11032-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.198.62:44411
--webserver_interface=127.10.198.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790376259650642-11032-0/krb5kdc/kudu_127.10.198.62.keytab
--principal=kudu/127.10.198.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.198.20:45719
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.198.62:44411
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790376259650642-11032-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790376259650642-11032-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790376259650642-11032-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:45:01.010550 19175 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:45:01.010906 19175 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:45:01.010952 19175 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:45:01.015452 19175 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 22:45:01.015597 19175 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:45:01.015631 19175 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:45:01.015658 19175 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 22:45:01.015688 19175 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 22:45:01.021718 19175 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.198.20:45719
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790376259650642-11032-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790376259650642-11032-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.198.62:44411
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790376259650642-11032-0/krb5kdc/kudu_127.10.198.62.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.198.62:44411
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790376259650642-11032-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.198.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19175
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790376259650642-11032-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:09:16 UTC on ccycloud.quasar-afmhyw.root.comops.site
build id 4673379
I20260925 22:45:01.022874 19175 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:45:01.023943 19175 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:45:01.033593 19180 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:45:01.033537 19181 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:45:01.033583 19183 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:45:01.034004 19175 server_base.cc:1034] running on GCE node
I20260925 22:45:01.034677 19175 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:45:01.036072 19175 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:45:01.037276 19175 hybrid_clock.cc:648] HybridClock initialized: now 1790376301037253 us; error 46 us; skew 500 ppm
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-09-25T22:45:01Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1359 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376259650642-11032-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 22:45:01 dist-test-slave-5vdr krb5kdc[19188](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 22:45:01 dist-test-slave-5vdr krb5kdc[19188](info): set up 2 sockets
Sep 25 22:45:01 dist-test-slave-5vdr krb5kdc[19188](info): commencing operation
krb5kdc: starting...
W20260925 22:45:02.105062 11032 mini_kdc.cc:121] Time spent starting KDC: real 1.034s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376259650642-11032-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376259650642-11032-0/krb5kdc/test-user.keytab.
Sep 25 22:45:02 dist-test-slave-5vdr krb5kdc[19188](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376302, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T22:45:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T22:45:02Z Disabled control of system clock
WARNING: no policy specified for kudu/127.10.198.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.198.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.198.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376259650642-11032-0/krb5kdc/kudu_127.10.198.62.keytab.
Entry for principal kudu/127.10.198.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376259650642-11032-0/krb5kdc/kudu_127.10.198.62.keytab.
I20260925 22:45:02.239017 11032 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrWXm8X/build/debug/bin/kudu
/tmp/dist-test-taskrWXm8X/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376259650642-11032-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376259650642-11032-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376259650642-11032-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376259650642-11032-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.198.62:32879
--webserver_interface=127.10.198.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376259650642-11032-0/krb5kdc/kudu_127.10.198.62.keytab
--principal=kudu/127.10.198.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.198.20:33781
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.198.62:32879
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376259650642-11032-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376259650642-11032-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376259650642-11032-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:45:02.362568 19201 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:45:02.362967 19201 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:45:02.363044 19201 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:45:02.367691 19201 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 22:45:02.367832 19201 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:45:02.367892 19201 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:45:02.367925 19201 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 22:45:02.367959 19201 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 22:45:02.374012 19201 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.198.20:33781
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376259650642-11032-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376259650642-11032-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.198.62:32879
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376259650642-11032-0/krb5kdc/kudu_127.10.198.62.keytab
--principal=kudu/127.10.198.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.198.62:32879
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376259650642-11032-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.198.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19201
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376259650642-11032-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:09:16 UTC on ccycloud.quasar-afmhyw.root.comops.site
build id 4673379
I20260925 22:45:02.375288 19201 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:45:02.376471 19201 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:45:02.385967 19207 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:45:02.385978 19209 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:45:02.386075 19201 server_base.cc:1034] running on GCE node
W20260925 22:45:02.385967 19206 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:45:02.386950 19201 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:45:02.388325 19201 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:45:02.389523 19201 hybrid_clock.cc:648] HybridClock initialized: now 1790376302389509 us; error 34 us; skew 500 ppm
Sep 25 22:45:02 dist-test-slave-5vdr krb5kdc[19188](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376302, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.198.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:45:02.392796 19201 init.cc:377] Logged in from keytab as kudu/127.10.198.62@KRBTEST.COM (short username kudu)
I20260925 22:45:02.393939 19201 webserver.cc:460] Webserver started at http://127.10.198.62:43519/ using document root <none> and password file <none>
I20260925 22:45:02.394670 19201 fs_manager.cc:359] Metadata directory not provided
I20260925 22:45:02.394748 19201 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:45:02.394994 19201 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:45:02.397444 19201 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376259650642-11032-0/minicluster-data/master-0/data/instance:
uuid: "b9a24ddf4fe34e618586688f4a9f09d7"
format_stamp: "Formatted at 2026-09-25 22:45:02 on dist-test-slave-5vdr"
server_key: "788785d77e635a7d63b3a86dc1a22897"
server_key_iv: "71a08b0e350e22459f7c1b5c577699ea"
server_key_version: "encryptionkey@0"
I20260925 22:45:02.398404 19201 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376259650642-11032-0/minicluster-data/master-0/wal/instance:
uuid: "b9a24ddf4fe34e618586688f4a9f09d7"
format_stamp: "Formatted at 2026-09-25 22:45:02 on dist-test-slave-5vdr"
server_key: "788785d77e635a7d63b3a86dc1a22897"
server_key_iv: "71a08b0e350e22459f7c1b5c577699ea"
server_key_version: "encryptionkey@0"
I20260925 22:45:02.403352 19201 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.004s
I20260925 22:45:02.406661 19216 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:45:02.408282 19201 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 22:45:02.408456 19201 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376259650642-11032-0/minicluster-data/master-0/data,/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376259650642-11032-0/minicluster-data/master-0/wal
uuid: "b9a24ddf4fe34e618586688f4a9f09d7"
format_stamp: "Formatted at 2026-09-25 22:45:02 on dist-test-slave-5vdr"
server_key: "788785d77e635a7d63b3a86dc1a22897"
server_key_iv: "71a08b0e350e22459f7c1b5c577699ea"
server_key_version: "encryptionkey@0"
I20260925 22:45:02.408600 19201 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376259650642-11032-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376259650642-11032-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376259650642-11032-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:45:02.453831 19201 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:45:02.457206 19201 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:45:02.457432 19201 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:45:02.465312 19201 rpc_server.cc:307] RPC server started. Bound to: 127.10.198.62:32879
I20260925 22:45:02.465322 19268 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.198.62:32879 every 8 connection(s)
I20260925 22:45:02.466426 19201 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376259650642-11032-0/minicluster-data/master-0/data/info.pb
I20260925 22:45:02.467499 11032 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrWXm8X/build/debug/bin/kudu as pid 19201
I20260925 22:45:02.467650 11032 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376259650642-11032-0/minicluster-data/master-0/wal/instance
I20260925 22:45:02.468032 11032 external_mini_cluster.cc:1428] Setting key 52adaffd5449705749998247eb8802bd
I20260925 22:45:02.470350 19269 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:45:02.476794 19269 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b9a24ddf4fe34e618586688f4a9f09d7: Bootstrap starting.
Sep 25 22:45:02 dist-test-slave-5vdr krb5kdc[19188](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376302, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.198.62@KRBTEST.COM
I20260925 22:45:02.479475 19269 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b9a24ddf4fe34e618586688f4a9f09d7: Neither blocks nor log segments found. Creating new log.
I20260925 22:45:02.480418 19269 log.cc:824] T 00000000000000000000000000000000 P b9a24ddf4fe34e618586688f4a9f09d7: Log is configured to *not* fsync() on all Append() calls
I20260925 22:45:02.482734 19269 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b9a24ddf4fe34e618586688f4a9f09d7: No bootstrap required, opened a new log
I20260925 22:45:02.483664 19272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:45:02.469586 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47826 (local address 127.10.198.62:32879)
0925 22:45:02.470190 (+ 604us) server_negotiation.cc:206] Beginning negotiation
0925 22:45:02.470209 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0925 22:45:02.470253 (+ 44us) server_negotiation.cc:407] Connection header received
0925 22:45:02.471003 (+ 750us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:45:02.471023 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:45:02.471466 (+ 443us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:45:02.471843 (+ 377us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:45:02.472746 (+ 903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:45:02.473815 (+ 1069us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:45:02.474763 (+ 948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:45:02.475307 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:45:02.475504 (+ 197us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:45:02.478295 (+ 2791us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:45:02.478300 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:45:02.478317 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:45:02.478355 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:45:02.480429 (+ 2074us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:45:02.480994 (+ 565us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:45:02.481000 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:45:02.481005 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:45:02.481131 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:45:02.481493 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:45:02.481496 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:45:02.481498 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:45:02.481954 (+ 456us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:45:02.482184 (+ 230us) server_negotiation.cc:1035] Waiting for connection context
0925 22:45:02.482466 (+ 282us) server_negotiation.cc:299] Negotiation successful
0925 22:45:02.482824 (+ 358us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":374,"thread_start_us":186,"threads_started":1}
I20260925 22:45:02.487112 19269 raft_consensus.cc:348] T 00000000000000000000000000000000 P b9a24ddf4fe34e618586688f4a9f09d7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b9a24ddf4fe34e618586688f4a9f09d7" member_type: VOTER last_known_addr { host: "127.10.198.62" port: 32879 } }
I20260925 22:45:02.487432 19269 raft_consensus.cc:374] T 00000000000000000000000000000000 P b9a24ddf4fe34e618586688f4a9f09d7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:45:02.487526 19269 raft_consensus.cc:729] T 00000000000000000000000000000000 P b9a24ddf4fe34e618586688f4a9f09d7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b9a24ddf4fe34e618586688f4a9f09d7, State: Initialized, Role: FOLLOWER
I20260925 22:45:02.488081 19269 consensus_queue.cc:260] T 00000000000000000000000000000000 P b9a24ddf4fe34e618586688f4a9f09d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b9a24ddf4fe34e618586688f4a9f09d7" member_type: VOTER last_known_addr { host: "127.10.198.62" port: 32879 } }
I20260925 22:45:02.488265 19269 raft_consensus.cc:388] T 00000000000000000000000000000000 P b9a24ddf4fe34e618586688f4a9f09d7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 22:45:02.488358 19269 raft_consensus.cc:482] T 00000000000000000000000000000000 P b9a24ddf4fe34e618586688f4a9f09d7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 22:45:02.488499 19269 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b9a24ddf4fe34e618586688f4a9f09d7 [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:45:02.489596 19269 raft_consensus.cc:504] T 00000000000000000000000000000000 P b9a24ddf4fe34e618586688f4a9f09d7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b9a24ddf4fe34e618586688f4a9f09d7" member_type: VOTER last_known_addr { host: "127.10.198.62" port: 32879 } }
I20260925 22:45:02.490013 19269 leader_election.cc:302] T 00000000000000000000000000000000 P b9a24ddf4fe34e618586688f4a9f09d7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b9a24ddf4fe34e618586688f4a9f09d7; no voters:
I20260925 22:45:02.490399 19269 leader_election.cc:288] T 00000000000000000000000000000000 P b9a24ddf4fe34e618586688f4a9f09d7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 22:45:02.490634 19274 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b9a24ddf4fe34e618586688f4a9f09d7 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 22:45:02.490962 19274 raft_consensus.cc:686] T 00000000000000000000000000000000 P b9a24ddf4fe34e618586688f4a9f09d7 [term 1 LEADER]: Becoming Leader. State: Replica: b9a24ddf4fe34e618586688f4a9f09d7, State: Running, Role: LEADER
I20260925 22:45:02.491654 19274 consensus_queue.cc:237] T 00000000000000000000000000000000 P b9a24ddf4fe34e618586688f4a9f09d7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b9a24ddf4fe34e618586688f4a9f09d7" member_type: VOTER last_known_addr { host: "127.10.198.62" port: 32879 } }
I20260925 22:45:02.492046 19269 sys_catalog.cc:564] T 00000000000000000000000000000000 P b9a24ddf4fe34e618586688f4a9f09d7 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 22:45:02.493527 19276 sys_catalog.cc:455] T 00000000000000000000000000000000 P b9a24ddf4fe34e618586688f4a9f09d7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b9a24ddf4fe34e618586688f4a9f09d7. Latest consensus state: current_term: 1 leader_uuid: "b9a24ddf4fe34e618586688f4a9f09d7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b9a24ddf4fe34e618586688f4a9f09d7" member_type: VOTER last_known_addr { host: "127.10.198.62" port: 32879 } } }
I20260925 22:45:02.493582 19275 sys_catalog.cc:455] T 00000000000000000000000000000000 P b9a24ddf4fe34e618586688f4a9f09d7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b9a24ddf4fe34e618586688f4a9f09d7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b9a24ddf4fe34e618586688f4a9f09d7" member_type: VOTER last_known_addr { host: "127.10.198.62" port: 32879 } } }
I20260925 22:45:02.493677 19276 sys_catalog.cc:458] T 00000000000000000000000000000000 P b9a24ddf4fe34e618586688f4a9f09d7 [sys.catalog]: This master's current role is: LEADER
I20260925 22:45:02.493716 19275 sys_catalog.cc:458] T 00000000000000000000000000000000 P b9a24ddf4fe34e618586688f4a9f09d7 [sys.catalog]: This master's current role is: LEADER
I20260925 22:45:02.495213 19285 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 22:45:02.498787 19285 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 22:45:02.506098 19285 catalog_manager.cc:1347] Generated new cluster ID: 1006a3e444ae4835b165910126b6b687
I20260925 22:45:02.506201 19285 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 22:45:02.528239 19285 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 22:45:02.529717 19285 catalog_manager.cc:1504] Loading token signing keys...
I20260925 22:45:02.535240 19285 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b9a24ddf4fe34e618586688f4a9f09d7: Generated new TSK 0
I20260925 22:45:02.536278 19285 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.10.198.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.198.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.198.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376259650642-11032-0/krb5kdc/kudu_127.10.198.1.keytab.
Entry for principal kudu/127.10.198.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376259650642-11032-0/krb5kdc/kudu_127.10.198.1.keytab.
I20260925 22:45:02.572281 11032 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrWXm8X/build/debug/bin/kudu
/tmp/dist-test-taskrWXm8X/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376259650642-11032-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376259650642-11032-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376259650642-11032-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376259650642-11032-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.198.1:0
--local_ip_for_outbound_sockets=127.10.198.1
--webserver_interface=127.10.198.1
--webserver_port=0
--tserver_master_addrs=127.10.198.62:32879
--keytab_file=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376259650642-11032-0/krb5kdc/kudu_127.10.198.1.keytab
--principal=kudu/127.10.198.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.198.20:33781
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376259650642-11032-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376259650642-11032-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376259650642-11032-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:45:02.697152 19295 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:45:02.697532 19295 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:45:02.697619 19295 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:45:02.702082 19295 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:45:02.702217 19295 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:45:02.702328 19295 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.198.1
I20260925 22:45:02.708293 19295 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.198.20:33781
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376259650642-11032-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376259650642-11032-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376259650642-11032-0/krb5kdc/kudu_127.10.198.1.keytab
--principal=kudu/127.10.198.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.198.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376259650642-11032-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.198.1
--webserver_port=0
--tserver_master_addrs=127.10.198.62:32879
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19295
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.198.1
--log_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376259650642-11032-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:09:16 UTC on ccycloud.quasar-afmhyw.root.comops.site
build id 4673379
I20260925 22:45:02.709604 19295 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:45:02.710800 19295 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 22:45:02.722182 19295 server_base.cc:1034] running on GCE node
W20260925 22:45:02.722136 19300 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:45:02.722158 19301 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:45:02.722136 19303 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:45:02.722939 19295 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:45:02.723654 19295 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:45:02.724861 19295 hybrid_clock.cc:648] HybridClock initialized: now 1790376302724831 us; error 38 us; skew 500 ppm
Sep 25 22:45:02 dist-test-slave-5vdr krb5kdc[19188](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376302, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.198.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:45:02.728101 19295 init.cc:377] Logged in from keytab as kudu/127.10.198.1@KRBTEST.COM (short username kudu)
I20260925 22:45:02.729207 19295 webserver.cc:460] Webserver started at http://127.10.198.1:44837/ using document root <none> and password file <none>
I20260925 22:45:02.729822 19295 fs_manager.cc:359] Metadata directory not provided
I20260925 22:45:02.729902 19295 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:45:02.730144 19295 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:45:02.732546 19295 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376259650642-11032-0/minicluster-data/ts-0/data/instance:
uuid: "5bc2477d037f4ca697494fc9a54f7b62"
format_stamp: "Formatted at 2026-09-25 22:45:02 on dist-test-slave-5vdr"
server_key: "ba6cd3041d783b97361858b2f63045af"
server_key_iv: "72e05f5aa61e67882c94099bf33bf65e"
server_key_version: "encryptionkey@0"
I20260925 22:45:02.733211 19295 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376259650642-11032-0/minicluster-data/ts-0/wal/instance:
uuid: "5bc2477d037f4ca697494fc9a54f7b62"
format_stamp: "Formatted at 2026-09-25 22:45:02 on dist-test-slave-5vdr"
server_key: "ba6cd3041d783b97361858b2f63045af"
server_key_iv: "72e05f5aa61e67882c94099bf33bf65e"
server_key_version: "encryptionkey@0"
I20260925 22:45:02.737577 19295 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260925 22:45:02.740520 19310 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:45:02.741982 19295 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 22:45:02.742141 19295 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376259650642-11032-0/minicluster-data/ts-0/data,/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376259650642-11032-0/minicluster-data/ts-0/wal
uuid: "5bc2477d037f4ca697494fc9a54f7b62"
format_stamp: "Formatted at 2026-09-25 22:45:02 on dist-test-slave-5vdr"
server_key: "ba6cd3041d783b97361858b2f63045af"
server_key_iv: "72e05f5aa61e67882c94099bf33bf65e"
server_key_version: "encryptionkey@0"
I20260925 22:45:02.742282 19295 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376259650642-11032-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376259650642-11032-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376259650642-11032-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:45:02.753777 19295 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:45:02.757553 19295 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:45:02.757798 19295 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:45:02.758469 19295 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 22:45:02.759379 19295 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:45:02.759429 19295 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:45:02.759508 19295 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:45:02.759541 19295 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 22:45:02.770043 19295 rpc_server.cc:307] RPC server started. Bound to: 127.10.198.1:41143
I20260925 22:45:02.770061 19423 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.198.1:41143 every 8 connection(s)
I20260925 22:45:02.771044 19295 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376259650642-11032-0/minicluster-data/ts-0/data/info.pb
I20260925 22:45:02.781028 11032 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrWXm8X/build/debug/bin/kudu as pid 19295
I20260925 22:45:02.781230 11032 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376259650642-11032-0/minicluster-data/ts-0/wal/instance
I20260925 22:45:02.781548 11032 external_mini_cluster.cc:1428] Setting key 9046f92e375211bd1c327298dc1a6f85
Sep 25 22:45:02 dist-test-slave-5vdr krb5kdc[19188](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376302, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.198.1@KRBTEST.COM for kudu/127.10.198.62@KRBTEST.COM
I20260925 22:45:02.786566 19272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:45:02.773084 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.198.1:56853 (local address 127.10.198.62:32879)
0925 22:45:02.773236 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0925 22:45:02.773239 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 22:45:02.774170 (+ 931us) server_negotiation.cc:407] Connection header received
0925 22:45:02.775597 (+ 1427us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:45:02.775601 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:45:02.775666 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:45:02.775744 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:45:02.777142 (+ 1398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:45:02.777720 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:45:02.779011 (+ 1291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:45:02.779274 (+ 263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:45:02.779422 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:45:02.782473 (+ 3051us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:45:02.782483 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:45:02.782485 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:45:02.782515 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:45:02.784157 (+ 1642us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:45:02.784799 (+ 642us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:45:02.784802 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:45:02.784803 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:45:02.784896 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:45:02.785298 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:45:02.785301 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:45:02.785303 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:45:02.785526 (+ 223us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:45:02.785644 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0925 22:45:02.786223 (+ 579us) server_negotiation.cc:299] Negotiation successful
0925 22:45:02.786389 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260925 22:45:02.787523 19426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:45:02.773473 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.198.62:32879 (local address 127.10.198.1:56853)
0925 22:45:02.773996 (+ 523us) negotiation.cc:107] Waiting for socket to connect
0925 22:45:02.774050 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0925 22:45:02.775333 (+ 1283us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:45:02.776003 (+ 670us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:45:02.776013 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:45:02.776426 (+ 413us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:45:02.776967 (+ 541us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:45:02.776982 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:45:02.777893 (+ 911us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:45:02.777896 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:45:02.778841 (+ 945us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:45:02.778852 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:45:02.779483 (+ 631us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:45:02.779487 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:45:02.779682 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:45:02.780446 (+ 764us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:45:02.780465 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:45:02.782244 (+ 1779us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:45:02.784316 (+ 2072us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:45:02.784324 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:45:02.784352 (+ 28us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:45:02.784663 (+ 311us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:45:02.785001 (+ 338us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:45:02.785004 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:45:02.785006 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:45:02.785182 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:45:02.785644 (+ 462us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:45:02.785654 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:45:02.785960 (+ 306us) client_negotiation.cc:769] Sending connection context
0925 22:45:02.786181 (+ 221us) client_negotiation.cc:240] Negotiation successful
0925 22:45:02.786563 (+ 382us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":302,"thread_start_us":127,"threads_started":1}
I20260925 22:45:02.789315 19424 heartbeater.cc:347] Connected to a master server at 127.10.198.62:32879
I20260925 22:45:02.789629 19424 heartbeater.cc:464] Registering TS with master...
I20260925 22:45:02.790233 19424 heartbeater.cc:511] Master 127.10.198.62:32879 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.10.198.2@KRBTEST.COM; defaulting to no policy
I20260925 22:45:02.792397 19233 ts_manager.cc:194] Registered new tserver with Master: 5bc2477d037f4ca697494fc9a54f7b62 (127.10.198.1:41143)
I20260925 22:45:02.793808 19233 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.198.1@KRBTEST.COM'} at 127.10.198.1:56853
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.198.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.198.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376259650642-11032-0/krb5kdc/kudu_127.10.198.2.keytab.
Entry for principal kudu/127.10.198.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376259650642-11032-0/krb5kdc/kudu_127.10.198.2.keytab.
I20260925 22:45:02.812779 11032 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrWXm8X/build/debug/bin/kudu
/tmp/dist-test-taskrWXm8X/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376259650642-11032-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376259650642-11032-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376259650642-11032-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376259650642-11032-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.198.2:0
--local_ip_for_outbound_sockets=127.10.198.2
--webserver_interface=127.10.198.2
--webserver_port=0
--tserver_master_addrs=127.10.198.62:32879
--keytab_file=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376259650642-11032-0/krb5kdc/kudu_127.10.198.2.keytab
--principal=kudu/127.10.198.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.198.20:33781
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376259650642-11032-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376259650642-11032-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376259650642-11032-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:45:02.934346 19429 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:45:02.934669 19429 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:45:02.934715 19429 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:45:02.939090 19429 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:45:02.939175 19429 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:45:02.939239 19429 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.198.2
I20260925 22:45:02.945089 19429 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.198.20:33781
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376259650642-11032-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376259650642-11032-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376259650642-11032-0/krb5kdc/kudu_127.10.198.2.keytab
--principal=kudu/127.10.198.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.198.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376259650642-11032-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.198.2
--webserver_port=0
--tserver_master_addrs=127.10.198.62:32879
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19429
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.198.2
--log_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376259650642-11032-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:09:16 UTC on ccycloud.quasar-afmhyw.root.comops.site
build id 4673379
I20260925 22:45:02.946329 19429 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:45:02.947522 19429 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:45:02.958535 19435 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:45:02.958561 19434 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:45:02.958561 19437 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:45:02.959059 19429 server_base.cc:1034] running on GCE node
I20260925 22:45:02.959605 19429 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:45:02.960469 19429 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:45:02.961699 19429 hybrid_clock.cc:648] HybridClock initialized: now 1790376302961686 us; error 42 us; skew 500 ppm
Sep 25 22:45:02 dist-test-slave-5vdr krb5kdc[19188](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376302, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.198.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:45:02.965297 19429 init.cc:377] Logged in from keytab as kudu/127.10.198.2@KRBTEST.COM (short username kudu)
I20260925 22:45:02.966355 19429 webserver.cc:460] Webserver started at http://127.10.198.2:39483/ using document root <none> and password file <none>
I20260925 22:45:02.966985 19429 fs_manager.cc:359] Metadata directory not provided
I20260925 22:45:02.967060 19429 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:45:02.967248 19429 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:45:02.969765 19429 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376259650642-11032-0/minicluster-data/ts-1/data/instance:
uuid: "5b1a0e3292234ad5bd61eb1e0d834157"
format_stamp: "Formatted at 2026-09-25 22:45:02 on dist-test-slave-5vdr"
server_key: "5f8d7c88d28053c7a95cc459b31c0351"
server_key_iv: "62677311ddd2dc526bfe2a30eef7a1c8"
server_key_version: "encryptionkey@0"
I20260925 22:45:02.970425 19429 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376259650642-11032-0/minicluster-data/ts-1/wal/instance:
uuid: "5b1a0e3292234ad5bd61eb1e0d834157"
format_stamp: "Formatted at 2026-09-25 22:45:02 on dist-test-slave-5vdr"
server_key: "5f8d7c88d28053c7a95cc459b31c0351"
server_key_iv: "62677311ddd2dc526bfe2a30eef7a1c8"
server_key_version: "encryptionkey@0"
I20260925 22:45:02.975116 19429 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260925 22:45:02.978426 19444 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:45:02.980180 19429 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 22:45:02.980341 19429 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376259650642-11032-0/minicluster-data/ts-1/data,/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376259650642-11032-0/minicluster-data/ts-1/wal
uuid: "5b1a0e3292234ad5bd61eb1e0d834157"
format_stamp: "Formatted at 2026-09-25 22:45:02 on dist-test-slave-5vdr"
server_key: "5f8d7c88d28053c7a95cc459b31c0351"
server_key_iv: "62677311ddd2dc526bfe2a30eef7a1c8"
server_key_version: "encryptionkey@0"
I20260925 22:45:02.980520 19429 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376259650642-11032-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376259650642-11032-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376259650642-11032-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:45:02.992869 19429 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:45:02.996842 19429 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:45:02.997077 19429 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:45:02.997715 19429 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 22:45:02.998636 19429 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:45:02.998713 19429 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:45:02.998785 19429 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:45:02.998831 19429 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 22:45:03.008878 19429 rpc_server.cc:307] RPC server started. Bound to: 127.10.198.2:41367
I20260925 22:45:03.008930 19557 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.198.2:41367 every 8 connection(s)
I20260925 22:45:03.009940 19429 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376259650642-11032-0/minicluster-data/ts-1/data/info.pb
I20260925 22:45:03.010648 11032 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrWXm8X/build/debug/bin/kudu as pid 19429
I20260925 22:45:03.010775 11032 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376259650642-11032-0/minicluster-data/ts-1/wal/instance
I20260925 22:45:03.011093 11032 external_mini_cluster.cc:1428] Setting key 75a756a2f8aa79ed8376ee739936297b
WARNING: no policy specified for kudu/127.10.198.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.198.3@KRBTEST.COM" created.
Sep 25 22:45:03 dist-test-slave-5vdr krb5kdc[19188](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376302, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.198.2@KRBTEST.COM for kudu/127.10.198.62@KRBTEST.COM
I20260925 22:45:03.031096 19272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:45:03.012202 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.198.2:40665 (local address 127.10.198.62:32879)
0925 22:45:03.012355 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0925 22:45:03.012359 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:45:03.013642 (+ 1283us) server_negotiation.cc:407] Connection header received
0925 22:45:03.015819 (+ 2177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:45:03.015825 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:45:03.015966 (+ 141us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:45:03.016078 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:45:03.017451 (+ 1373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:45:03.018370 (+ 919us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:45:03.019743 (+ 1373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:45:03.020198 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:45:03.020340 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:45:03.026170 (+ 5830us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:45:03.026174 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:45:03.026177 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:45:03.026210 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:45:03.028180 (+ 1970us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:45:03.028986 (+ 806us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:45:03.028989 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:45:03.028990 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:45:03.029082 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:45:03.029583 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:45:03.029585 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:45:03.029586 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:45:03.029865 (+ 279us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:45:03.029949 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0925 22:45:03.030808 (+ 859us) server_negotiation.cc:299] Negotiation successful
0925 22:45:03.030951 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260925 22:45:03.032740 19560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:45:03.012786 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.198.62:32879 (local address 127.10.198.2:40665)
0925 22:45:03.013441 (+ 655us) negotiation.cc:107] Waiting for socket to connect
0925 22:45:03.013525 (+ 84us) client_negotiation.cc:174] Beginning negotiation
0925 22:45:03.015512 (+ 1987us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:45:03.016275 (+ 763us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:45:03.016283 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:45:03.016689 (+ 406us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:45:03.017257 (+ 568us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:45:03.017271 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:45:03.018559 (+ 1288us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:45:03.018571 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:45:03.019569 (+ 998us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:45:03.019582 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:45:03.020372 (+ 790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:45:03.020375 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:45:03.020549 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:45:03.021258 (+ 709us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:45:03.021276 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:45:03.025880 (+ 4604us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:45:03.028355 (+ 2475us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:45:03.028367 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:45:03.028393 (+ 26us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:45:03.028849 (+ 456us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:45:03.029203 (+ 354us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:45:03.029209 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:45:03.029214 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:45:03.029455 (+ 241us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:45:03.030021 (+ 566us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:45:03.030033 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:45:03.030451 (+ 418us) client_negotiation.cc:769] Sending connection context
0925 22:45:03.030733 (+ 282us) client_negotiation.cc:240] Negotiation successful
0925 22:45:03.031208 (+ 475us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":356,"thread_start_us":149,"threads_started":1}
I20260925 22:45:03.034314 19558 heartbeater.cc:347] Connected to a master server at 127.10.198.62:32879
I20260925 22:45:03.034646 19558 heartbeater.cc:464] Registering TS with master...
I20260925 22:45:03.035354 19558 heartbeater.cc:511] Master 127.10.198.62:32879 requested a full tablet report, sending...
I20260925 22:45:03.036921 19233 ts_manager.cc:194] Registered new tserver with Master: 5b1a0e3292234ad5bd61eb1e0d834157 (127.10.198.2:41367)
I20260925 22:45:03.037650 19233 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.198.2@KRBTEST.COM'} at 127.10.198.2:40665
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.198.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376259650642-11032-0/krb5kdc/kudu_127.10.198.3.keytab.
Entry for principal kudu/127.10.198.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376259650642-11032-0/krb5kdc/kudu_127.10.198.3.keytab.
I20260925 22:45:03.043795 11032 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrWXm8X/build/debug/bin/kudu
/tmp/dist-test-taskrWXm8X/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376259650642-11032-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376259650642-11032-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376259650642-11032-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376259650642-11032-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.198.3:0
--local_ip_for_outbound_sockets=127.10.198.3
--webserver_interface=127.10.198.3
--webserver_port=0
--tserver_master_addrs=127.10.198.62:32879
--keytab_file=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376259650642-11032-0/krb5kdc/kudu_127.10.198.3.keytab
--principal=kudu/127.10.198.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.198.20:33781
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376259650642-11032-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376259650642-11032-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376259650642-11032-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:45:03.167438 19563 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:45:03.167776 19563 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:45:03.167831 19563 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:45:03.172478 19563 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:45:03.172581 19563 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:45:03.172652 19563 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.198.3
I20260925 22:45:03.178543 19563 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.198.20:33781
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376259650642-11032-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376259650642-11032-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376259650642-11032-0/krb5kdc/kudu_127.10.198.3.keytab
--principal=kudu/127.10.198.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.198.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376259650642-11032-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.198.3
--webserver_port=0
--tserver_master_addrs=127.10.198.62:32879
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19563
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.198.3
--log_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376259650642-11032-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:09:16 UTC on ccycloud.quasar-afmhyw.root.comops.site
build id 4673379
I20260925 22:45:03.179733 19563 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:45:03.180837 19563 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:45:03.192095 19568 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:45:03.192093 19571 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:45:03.192096 19569 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:45:03.192413 19563 server_base.cc:1034] running on GCE node
I20260925 22:45:03.193115 19563 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:45:03.193892 19563 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:45:03.195111 19563 hybrid_clock.cc:648] HybridClock initialized: now 1790376303195094 us; error 81 us; skew 500 ppm
Sep 25 22:45:03 dist-test-slave-5vdr krb5kdc[19188](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376303, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.198.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:45:03.198462 19563 init.cc:377] Logged in from keytab as kudu/127.10.198.3@KRBTEST.COM (short username kudu)
I20260925 22:45:03.199746 19563 webserver.cc:460] Webserver started at http://127.10.198.3:42697/ using document root <none> and password file <none>
I20260925 22:45:03.200538 19563 fs_manager.cc:359] Metadata directory not provided
I20260925 22:45:03.200670 19563 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:45:03.200927 19563 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:45:03.203541 19563 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376259650642-11032-0/minicluster-data/ts-2/data/instance:
uuid: "fe69c4947a5b4d6eabf4749c51867ffb"
format_stamp: "Formatted at 2026-09-25 22:45:03 on dist-test-slave-5vdr"
server_key: "8fd08cea63366d85477442881b0c7893"
server_key_iv: "e507616b1b02b3dbdf7ec14dca6945b2"
server_key_version: "encryptionkey@0"
I20260925 22:45:03.204627 19563 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376259650642-11032-0/minicluster-data/ts-2/wal/instance:
uuid: "fe69c4947a5b4d6eabf4749c51867ffb"
format_stamp: "Formatted at 2026-09-25 22:45:03 on dist-test-slave-5vdr"
server_key: "8fd08cea63366d85477442881b0c7893"
server_key_iv: "e507616b1b02b3dbdf7ec14dca6945b2"
server_key_version: "encryptionkey@0"
I20260925 22:45:03.209262 19563 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260925 22:45:03.213171 19578 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:45:03.214723 19563 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260925 22:45:03.214898 19563 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376259650642-11032-0/minicluster-data/ts-2/data,/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376259650642-11032-0/minicluster-data/ts-2/wal
uuid: "fe69c4947a5b4d6eabf4749c51867ffb"
format_stamp: "Formatted at 2026-09-25 22:45:03 on dist-test-slave-5vdr"
server_key: "8fd08cea63366d85477442881b0c7893"
server_key_iv: "e507616b1b02b3dbdf7ec14dca6945b2"
server_key_version: "encryptionkey@0"
I20260925 22:45:03.215050 19563 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376259650642-11032-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376259650642-11032-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376259650642-11032-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:45:03.233373 19563 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:45:03.237097 19563 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:45:03.237293 19563 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:45:03.238045 19563 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 22:45:03.239604 19563 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:45:03.239708 19563 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:45:03.239780 19563 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:45:03.239836 19563 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 22:45:03.249917 19563 rpc_server.cc:307] RPC server started. Bound to: 127.10.198.3:42213
I20260925 22:45:03.249994 19691 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.198.3:42213 every 8 connection(s)
I20260925 22:45:03.251013 19563 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376259650642-11032-0/minicluster-data/ts-2/data/info.pb
I20260925 22:45:03.252401 11032 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrWXm8X/build/debug/bin/kudu as pid 19563
I20260925 22:45:03.252556 11032 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376259650642-11032-0/minicluster-data/ts-2/wal/instance
I20260925 22:45:03.252910 11032 external_mini_cluster.cc:1428] Setting key a5faa6c0491c47af6d5e68a2312652b9
Sep 25 22:45:03 dist-test-slave-5vdr krb5kdc[19188](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376303, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.198.3@KRBTEST.COM for kudu/127.10.198.62@KRBTEST.COM
I20260925 22:45:03.268308 19272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:45:03.253657 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.198.3:34837 (local address 127.10.198.62:32879)
0925 22:45:03.253825 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0925 22:45:03.253830 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:45:03.255011 (+ 1181us) server_negotiation.cc:407] Connection header received
0925 22:45:03.256680 (+ 1669us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:45:03.256684 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:45:03.256752 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:45:03.256847 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:45:03.258306 (+ 1459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:45:03.258917 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:45:03.260574 (+ 1657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:45:03.260818 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:45:03.260948 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:45:03.263772 (+ 2824us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:45:03.263775 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:45:03.263777 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:45:03.263808 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:45:03.265608 (+ 1800us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:45:03.266316 (+ 708us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:45:03.266320 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:45:03.266322 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:45:03.266434 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:45:03.266922 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:45:03.266925 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:45:03.266927 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:45:03.267213 (+ 286us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:45:03.267321 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0925 22:45:03.268036 (+ 715us) server_negotiation.cc:299] Negotiation successful
0925 22:45:03.268172 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260925 22:45:03.269213 19694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:45:03.254199 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.198.62:32879 (local address 127.10.198.3:34837)
0925 22:45:03.254849 (+ 650us) negotiation.cc:107] Waiting for socket to connect
0925 22:45:03.254902 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0925 22:45:03.256408 (+ 1506us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:45:03.257058 (+ 650us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:45:03.257067 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:45:03.257507 (+ 440us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:45:03.258103 (+ 596us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:45:03.258120 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:45:03.259093 (+ 973us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:45:03.259097 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:45:03.260397 (+ 1300us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:45:03.260410 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:45:03.261002 (+ 592us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:45:03.261008 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:45:03.261216 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:45:03.261883 (+ 667us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:45:03.261902 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:45:03.263536 (+ 1634us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:45:03.265775 (+ 2239us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:45:03.265790 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:45:03.265808 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:45:03.266192 (+ 384us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:45:03.266587 (+ 395us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:45:03.266590 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:45:03.266592 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:45:03.266806 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:45:03.267352 (+ 546us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:45:03.267361 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:45:03.267728 (+ 367us) client_negotiation.cc:769] Sending connection context
0925 22:45:03.267969 (+ 241us) client_negotiation.cc:240] Negotiation successful
0925 22:45:03.268286 (+ 317us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":419,"thread_start_us":180,"threads_started":1}
I20260925 22:45:03.270485 19692 heartbeater.cc:347] Connected to a master server at 127.10.198.62:32879
I20260925 22:45:03.270776 19692 heartbeater.cc:464] Registering TS with master...
I20260925 22:45:03.271339 19692 heartbeater.cc:511] Master 127.10.198.62:32879 requested a full tablet report, sending...
I20260925 22:45:03.272603 19233 ts_manager.cc:194] Registered new tserver with Master: fe69c4947a5b4d6eabf4749c51867ffb (127.10.198.3:42213)
I20260925 22:45:03.273224 19233 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.198.3@KRBTEST.COM'} at 127.10.198.3:34837
I20260925 22:45:03.279618 11032 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.10.198.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.10.198.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.10.198.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376259650642-11032-0/krb5kdc/oryx_127.10.198.1.keytab.
Entry for principal oryx/127.10.198.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376259650642-11032-0/krb5kdc/oryx_127.10.198.1.keytab.
WARNING: no policy specified for oryx/127.10.198.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.10.198.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.10.198.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376259650642-11032-0/krb5kdc/oryx_127.10.198.2.keytab.
Entry for principal oryx/127.10.198.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376259650642-11032-0/krb5kdc/oryx_127.10.198.2.keytab.
WARNING: no policy specified for oryx/127.10.198.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.10.198.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.10.198.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376259650642-11032-0/krb5kdc/oryx_127.10.198.3.keytab.
Entry for principal oryx/127.10.198.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376259650642-11032-0/krb5kdc/oryx_127.10.198.3.keytab.
I20260925 22:45:03.363040 11032 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrWXm8X/build/debug/bin/kudu with pid 19295
I20260925 22:45:03.368841 11032 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrWXm8X/build/debug/bin/kudu with pid 19429
I20260925 22:45:03.374807 11032 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrWXm8X/build/debug/bin/kudu with pid 19563
I20260925 22:45:03.380805 11032 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrWXm8X/build/debug/bin/kudu with pid 19201
I20260925 22:45:03.387789 11032 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrWXm8X/build/debug/bin/kudu
/tmp/dist-test-taskrWXm8X/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376259650642-11032-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376259650642-11032-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376259650642-11032-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376259650642-11032-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.198.62:32879
--webserver_interface=127.10.198.62
--webserver_port=43519
--keytab_file=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376259650642-11032-0/krb5kdc/kudu_127.10.198.62.keytab
--principal=kudu/127.10.198.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.198.20:33781
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.198.62:32879
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376259650642-11032-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376259650642-11032-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376259650642-11032-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:45:03.506938 19701 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:45:03.507349 19701 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:45:03.507434 19701 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:45:03.512120 19701 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 22:45:03.512229 19701 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:45:03.512265 19701 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:45:03.512331 19701 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 22:45:03.512362 19701 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 22:45:03.519502 19701 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.198.20:33781
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376259650642-11032-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376259650642-11032-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.198.62:32879
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376259650642-11032-0/krb5kdc/kudu_127.10.198.62.keytab
--principal=kudu/127.10.198.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.198.62:32879
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376259650642-11032-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.198.62
--webserver_port=43519
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19701
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376259650642-11032-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:09:16 UTC on ccycloud.quasar-afmhyw.root.comops.site
build id 4673379
I20260925 22:45:03.521235 19701 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:45:03.522819 19701 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:45:03.533128 19707 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:45:03.533128 19706 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:45:03.533118 19709 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:45:03.533536 19701 server_base.cc:1034] running on GCE node
I20260925 22:45:03.534202 19701 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:45:03.535475 19701 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:45:03.536664 19701 hybrid_clock.cc:648] HybridClock initialized: now 1790376303536623 us; error 64 us; skew 500 ppm
Sep 25 22:45:03 dist-test-slave-5vdr krb5kdc[19188](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376303, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.198.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:45:03.539783 19701 init.cc:377] Logged in from keytab as kudu/127.10.198.62@KRBTEST.COM (short username kudu)
I20260925 22:45:03.540953 19701 webserver.cc:460] Webserver started at http://127.10.198.62:43519/ using document root <none> and password file <none>
I20260925 22:45:03.541581 19701 fs_manager.cc:359] Metadata directory not provided
I20260925 22:45:03.541666 19701 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:45:03.546185 19701 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260925 22:45:03.548877 19716 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:45:03.550511 19701 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260925 22:45:03.550709 19701 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376259650642-11032-0/minicluster-data/master-0/data,/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376259650642-11032-0/minicluster-data/master-0/wal
uuid: "b9a24ddf4fe34e618586688f4a9f09d7"
format_stamp: "Formatted at 2026-09-25 22:45:02 on dist-test-slave-5vdr"
server_key: "788785d77e635a7d63b3a86dc1a22897"
server_key_iv: "71a08b0e350e22459f7c1b5c577699ea"
server_key_version: "encryptionkey@0"
I20260925 22:45:03.551366 19701 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376259650642-11032-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376259650642-11032-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376259650642-11032-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:45:03.575131 19701 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:45:03.578707 19701 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:45:03.578975 19701 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:45:03.587270 19701 rpc_server.cc:307] RPC server started. Bound to: 127.10.198.62:32879
I20260925 22:45:03.587301 19768 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.198.62:32879 every 8 connection(s)
I20260925 22:45:03.588526 19701 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376259650642-11032-0/minicluster-data/master-0/data/info.pb
I20260925 22:45:03.593410 19769 sys_catalog.cc:263] Verifying existing consensus state
I20260925 22:45:03.595439 11032 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrWXm8X/build/debug/bin/kudu as pid 19701
I20260925 22:45:03.595448 19769 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b9a24ddf4fe34e618586688f4a9f09d7: Bootstrap starting.
I20260925 22:45:03.596055 11032 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrWXm8X/build/debug/bin/kudu
/tmp/dist-test-taskrWXm8X/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376259650642-11032-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376259650642-11032-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376259650642-11032-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376259650642-11032-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.198.1:41143
--local_ip_for_outbound_sockets=127.10.198.1
--tserver_master_addrs=127.10.198.62:32879
--webserver_port=44837
--webserver_interface=127.10.198.1
--keytab_file=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376259650642-11032-0/krb5kdc/kudu_127.10.198.1.keytab
--principal=kudu/127.10.198.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.198.20:33781
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.10.198.1
--keytab_file=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376259650642-11032-0/krb5kdc/oryx_127.10.198.1.keytab with env {KRB5CCNAME=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376259650642-11032-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376259650642-11032-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376259650642-11032-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 22:45:03.615443 19769 log.cc:824] T 00000000000000000000000000000000 P b9a24ddf4fe34e618586688f4a9f09d7: Log is configured to *not* fsync() on all Append() calls
I20260925 22:45:03.624415 19769 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b9a24ddf4fe34e618586688f4a9f09d7: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 22:45:03.624961 19769 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b9a24ddf4fe34e618586688f4a9f09d7: Bootstrap complete.
I20260925 22:45:03.630178 19769 raft_consensus.cc:348] T 00000000000000000000000000000000 P b9a24ddf4fe34e618586688f4a9f09d7 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b9a24ddf4fe34e618586688f4a9f09d7" member_type: VOTER last_known_addr { host: "127.10.198.62" port: 32879 } }
I20260925 22:45:03.630798 19769 raft_consensus.cc:729] T 00000000000000000000000000000000 P b9a24ddf4fe34e618586688f4a9f09d7 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: b9a24ddf4fe34e618586688f4a9f09d7, State: Initialized, Role: FOLLOWER
I20260925 22:45:03.631422 19769 consensus_queue.cc:260] T 00000000000000000000000000000000 P b9a24ddf4fe34e618586688f4a9f09d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b9a24ddf4fe34e618586688f4a9f09d7" member_type: VOTER last_known_addr { host: "127.10.198.62" port: 32879 } }
I20260925 22:45:03.631633 19769 raft_consensus.cc:388] T 00000000000000000000000000000000 P b9a24ddf4fe34e618586688f4a9f09d7 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 22:45:03.631758 19769 raft_consensus.cc:482] T 00000000000000000000000000000000 P b9a24ddf4fe34e618586688f4a9f09d7 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 22:45:03.631951 19769 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b9a24ddf4fe34e618586688f4a9f09d7 [term 1 FOLLOWER]: Advancing to term 2
I20260925 22:45:03.634510 19769 raft_consensus.cc:504] T 00000000000000000000000000000000 P b9a24ddf4fe34e618586688f4a9f09d7 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b9a24ddf4fe34e618586688f4a9f09d7" member_type: VOTER last_known_addr { host: "127.10.198.62" port: 32879 } }
I20260925 22:45:03.634986 19769 leader_election.cc:302] T 00000000000000000000000000000000 P b9a24ddf4fe34e618586688f4a9f09d7 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b9a24ddf4fe34e618586688f4a9f09d7; no voters:
I20260925 22:45:03.635501 19769 leader_election.cc:288] T 00000000000000000000000000000000 P b9a24ddf4fe34e618586688f4a9f09d7 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260925 22:45:03.635640 19773 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b9a24ddf4fe34e618586688f4a9f09d7 [term 2 FOLLOWER]: Leader election won for term 2
I20260925 22:45:03.635928 19773 raft_consensus.cc:686] T 00000000000000000000000000000000 P b9a24ddf4fe34e618586688f4a9f09d7 [term 2 LEADER]: Becoming Leader. State: Replica: b9a24ddf4fe34e618586688f4a9f09d7, State: Running, Role: LEADER
I20260925 22:45:03.636314 19773 consensus_queue.cc:237] T 00000000000000000000000000000000 P b9a24ddf4fe34e618586688f4a9f09d7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b9a24ddf4fe34e618586688f4a9f09d7" member_type: VOTER last_known_addr { host: "127.10.198.62" port: 32879 } }
I20260925 22:45:03.637130 19769 sys_catalog.cc:564] T 00000000000000000000000000000000 P b9a24ddf4fe34e618586688f4a9f09d7 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 22:45:03.637946 19774 sys_catalog.cc:455] T 00000000000000000000000000000000 P b9a24ddf4fe34e618586688f4a9f09d7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "b9a24ddf4fe34e618586688f4a9f09d7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b9a24ddf4fe34e618586688f4a9f09d7" member_type: VOTER last_known_addr { host: "127.10.198.62" port: 32879 } } }
I20260925 22:45:03.638106 19774 sys_catalog.cc:458] T 00000000000000000000000000000000 P b9a24ddf4fe34e618586688f4a9f09d7 [sys.catalog]: This master's current role is: LEADER
I20260925 22:45:03.639110 19775 sys_catalog.cc:455] T 00000000000000000000000000000000 P b9a24ddf4fe34e618586688f4a9f09d7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b9a24ddf4fe34e618586688f4a9f09d7. Latest consensus state: current_term: 2 leader_uuid: "b9a24ddf4fe34e618586688f4a9f09d7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b9a24ddf4fe34e618586688f4a9f09d7" member_type: VOTER last_known_addr { host: "127.10.198.62" port: 32879 } } }
I20260925 22:45:03.639273 19775 sys_catalog.cc:458] T 00000000000000000000000000000000 P b9a24ddf4fe34e618586688f4a9f09d7 [sys.catalog]: This master's current role is: LEADER
I20260925 22:45:03.639729 19780 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 22:45:03.643648 19780 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 22:45:03.644521 19780 catalog_manager.cc:1259] Loaded cluster ID: 1006a3e444ae4835b165910126b6b687
I20260925 22:45:03.644636 19780 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 22:45:03.647408 19780 catalog_manager.cc:1504] Loading token signing keys...
I20260925 22:45:03.648451 19780 catalog_manager.cc:5879] T 00000000000000000000000000000000 P b9a24ddf4fe34e618586688f4a9f09d7: Loaded TSK: 0
I20260925 22:45:03.650003 19780 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260925 22:45:03.725871 19771 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:45:03.726248 19771 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:45:03.726334 19771 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:45:03.730901 19771 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:45:03.731030 19771 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:45:03.731137 19771 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.198.1
I20260925 22:45:03.737353 19771 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.198.20:33781
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376259650642-11032-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376259650642-11032-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376259650642-11032-0/krb5kdc/oryx_127.10.198.1.keytab
--principal=oryx/127.10.198.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.198.1:41143
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376259650642-11032-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.198.1
--webserver_port=44837
--tserver_master_addrs=127.10.198.62:32879
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19771
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.198.1
--log_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376259650642-11032-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:09:16 UTC on ccycloud.quasar-afmhyw.root.comops.site
build id 4673379
I20260925 22:45:03.738700 19771 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:45:03.740005 19771 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:45:03.752139 19795 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:45:03.752139 19796 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:45:03.752514 19798 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:45:03.754187 19771 server_base.cc:1034] running on GCE node
I20260925 22:45:03.754801 19771 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:45:03.755782 19771 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:45:03.757266 19771 hybrid_clock.cc:648] HybridClock initialized: now 1790376303757228 us; error 86 us; skew 500 ppm
Sep 25 22:45:03 dist-test-slave-5vdr krb5kdc[19188](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376303, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.198.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:45:03.762279 19771 init.cc:377] Logged in from keytab as oryx/127.10.198.1@KRBTEST.COM (short username oryx)
I20260925 22:45:03.763887 19771 webserver.cc:460] Webserver started at http://127.10.198.1:44837/ using document root <none> and password file <none>
I20260925 22:45:03.764711 19771 fs_manager.cc:359] Metadata directory not provided
I20260925 22:45:03.764844 19771 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:45:03.770970 19771 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260925 22:45:03.774307 19805 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:45:03.776599 19771 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260925 22:45:03.776785 19771 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376259650642-11032-0/minicluster-data/ts-0/data,/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376259650642-11032-0/minicluster-data/ts-0/wal
uuid: "5bc2477d037f4ca697494fc9a54f7b62"
format_stamp: "Formatted at 2026-09-25 22:45:02 on dist-test-slave-5vdr"
server_key: "ba6cd3041d783b97361858b2f63045af"
server_key_iv: "72e05f5aa61e67882c94099bf33bf65e"
server_key_version: "encryptionkey@0"
I20260925 22:45:03.777637 19771 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376259650642-11032-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376259650642-11032-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376259650642-11032-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:45:03.794019 19771 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:45:03.798540 19771 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:45:03.798789 19771 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:45:03.799644 19771 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 22:45:03.800851 19771 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:45:03.800940 19771 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:45:03.800999 19771 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:45:03.801043 19771 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 22:45:03.813432 19771 rpc_server.cc:307] RPC server started. Bound to: 127.10.198.1:41143
I20260925 22:45:03.813467 19918 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.198.1:41143 every 8 connection(s)
I20260925 22:45:03.814877 19771 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376259650642-11032-0/minicluster-data/ts-0/data/info.pb
I20260925 22:45:03.824462 11032 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrWXm8X/build/debug/bin/kudu as pid 19771
I20260925 22:45:03.825022 11032 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrWXm8X/build/debug/bin/kudu
/tmp/dist-test-taskrWXm8X/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376259650642-11032-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376259650642-11032-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376259650642-11032-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376259650642-11032-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.198.2:41367
--local_ip_for_outbound_sockets=127.10.198.2
--tserver_master_addrs=127.10.198.62:32879
--webserver_port=39483
--webserver_interface=127.10.198.2
--keytab_file=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376259650642-11032-0/krb5kdc/kudu_127.10.198.2.keytab
--principal=kudu/127.10.198.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.198.20:33781
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.10.198.2
--keytab_file=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376259650642-11032-0/krb5kdc/oryx_127.10.198.2.keytab with env {KRB5CCNAME=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376259650642-11032-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376259650642-11032-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376259650642-11032-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 25 22:45:03 dist-test-slave-5vdr krb5kdc[19188](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.198.1@KRBTEST.COM for oryx/127.10.198.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 22:45:03 dist-test-slave-5vdr krb5kdc[19188](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.198.1@KRBTEST.COM for oryx/127.10.198.62@KRBTEST.COM, Server not found in Kerberos database
W20260925 22:45:03.831493 19921 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:45:03.817475 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.198.1:55921 (local address 127.10.198.62:32879)
0925 22:45:03.817941 (+ 466us) server_negotiation.cc:206] Beginning negotiation
0925 22:45:03.817956 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0925 22:45:03.818720 (+ 764us) server_negotiation.cc:407] Connection header received
0925 22:45:03.819981 (+ 1261us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:45:03.819993 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:45:03.820339 (+ 346us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:45:03.820605 (+ 266us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:45:03.822261 (+ 1656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:45:03.823268 (+ 1007us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:45:03.824665 (+ 1397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:45:03.825545 (+ 880us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:45:03.825793 (+ 248us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:45:03.830563 (+ 4770us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.198.1:55921: BlockingRecv error: recv got EOF from 127.10.198.1:55921 (error 108)
Metrics: {"server-negotiator.queue_time_us":302,"thread_start_us":131,"threads_started":1}
I20260925 22:45:03.831688 19922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:45:03.817748 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.198.62:32879 (local address 127.10.198.1:55921)
0925 22:45:03.818536 (+ 788us) negotiation.cc:107] Waiting for socket to connect
0925 22:45:03.818583 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0925 22:45:03.819461 (+ 878us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:45:03.821125 (+ 1664us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:45:03.821132 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:45:03.821539 (+ 407us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:45:03.822048 (+ 509us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:45:03.822062 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:45:03.823443 (+ 1381us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:45:03.823446 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:45:03.824464 (+ 1018us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:45:03.824476 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:45:03.825786 (+ 1310us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:45:03.825789 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:45:03.825953 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:45:03.826695 (+ 742us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:45:03.826714 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:45:03.830256 (+ 3542us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.198.62:32879: Server oryx/127.10.198.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":582,"thread_start_us":169,"threads_started":1}
W20260925 22:45:03.832327 19919 heartbeater.cc:644] Failed to heartbeat to 127.10.198.62:32879 (0 consecutive failures): Not authorized: Failed to ping master at 127.10.198.62:32879: Client connection negotiation failed: client connection to 127.10.198.62:32879: Server oryx/127.10.198.62@KRBTEST.COM not found in Kerberos database
Sep 25 22:45:03 dist-test-slave-5vdr krb5kdc[19188](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.198.1@KRBTEST.COM for oryx/127.10.198.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 22:45:03 dist-test-slave-5vdr krb5kdc[19188](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.198.1@KRBTEST.COM for oryx/127.10.198.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 22:45:03.839497 19922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:45:03.832847 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.198.62:32879 (local address 127.10.198.1:50749)
0925 22:45:03.833022 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0925 22:45:03.833045 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0925 22:45:03.833180 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:45:03.833485 (+ 305us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:45:03.833490 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:45:03.833709 (+ 219us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:45:03.833881 (+ 172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:45:03.833892 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:45:03.834686 (+ 794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:45:03.834691 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:45:03.835763 (+ 1072us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:45:03.835779 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:45:03.836308 (+ 529us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:45:03.836315 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:45:03.836454 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:45:03.836996 (+ 542us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:45:03.837019 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:45:03.839328 (+ 2309us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.198.62:32879: Server oryx/127.10.198.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":62}
W20260925 22:45:03.839723 19921 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:45:03.832907 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.198.1:50749 (local address 127.10.198.62:32879)
0925 22:45:03.833028 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0925 22:45:03.833032 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:45:03.833129 (+ 97us) server_negotiation.cc:407] Connection header received
0925 22:45:03.833287 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:45:03.833289 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:45:03.833341 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:45:03.833415 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:45:03.834022 (+ 607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:45:03.834556 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:45:03.835941 (+ 1385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:45:03.836171 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:45:03.836255 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:45:03.839572 (+ 3317us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.198.1:50749: BlockingRecv error: recv got EOF from 127.10.198.1:50749 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
Sep 25 22:45:03 dist-test-slave-5vdr krb5kdc[19188](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.198.1@KRBTEST.COM for oryx/127.10.198.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 22:45:03 dist-test-slave-5vdr krb5kdc[19188](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.198.1@KRBTEST.COM for oryx/127.10.198.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 22:45:03.846442 19922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:45:03.840431 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.198.62:32879 (local address 127.10.198.1:56971)
0925 22:45:03.840555 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0925 22:45:03.840571 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 22:45:03.840658 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:45:03.840908 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:45:03.840910 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:45:03.841069 (+ 159us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:45:03.841160 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:45:03.841165 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:45:03.842282 (+ 1117us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:45:03.842285 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:45:03.842841 (+ 556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:45:03.842851 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:45:03.843462 (+ 611us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:45:03.843465 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:45:03.843531 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:45:03.844167 (+ 636us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:45:03.844181 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:45:03.846298 (+ 2117us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.198.62:32879: Server oryx/127.10.198.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":45}
W20260925 22:45:03.846709 19919 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260925 22:45:03.846916 19921 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:45:03.840515 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.198.1:56971 (local address 127.10.198.62:32879)
0925 22:45:03.840643 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0925 22:45:03.840647 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:45:03.840662 (+ 15us) server_negotiation.cc:407] Connection header received
0925 22:45:03.840751 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:45:03.840754 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:45:03.840804 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:45:03.840883 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:45:03.841276 (+ 393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:45:03.842122 (+ 846us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:45:03.842969 (+ 847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:45:03.843331 (+ 362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:45:03.843435 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:45:03.846523 (+ 3088us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.198.1:56971: BlockingRecv error: recv got EOF from 127.10.198.1:56971 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
W20260925 22:45:03.951707 19923 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:45:03.952124 19923 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:45:03.952178 19923 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:45:03.956614 19923 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:45:03.956694 19923 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:45:03.956761 19923 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.198.2
I20260925 22:45:03.962569 19923 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.198.20:33781
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376259650642-11032-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376259650642-11032-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376259650642-11032-0/krb5kdc/oryx_127.10.198.2.keytab
--principal=oryx/127.10.198.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.198.2:41367
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376259650642-11032-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.198.2
--webserver_port=39483
--tserver_master_addrs=127.10.198.62:32879
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19923
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.198.2
--log_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376259650642-11032-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:09:16 UTC on ccycloud.quasar-afmhyw.root.comops.site
build id 4673379
I20260925 22:45:03.963722 19923 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:45:03.964887 19923 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:45:03.975762 19931 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:45:03.975764 19929 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:45:03.975764 19928 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:45:03.976200 19923 server_base.cc:1034] running on GCE node
I20260925 22:45:03.976742 19923 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:45:03.977543 19923 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:45:03.978739 19923 hybrid_clock.cc:648] HybridClock initialized: now 1790376303978720 us; error 37 us; skew 500 ppm
Sep 25 22:45:03 dist-test-slave-5vdr krb5kdc[19188](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376303, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.198.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:45:03.981853 19923 init.cc:377] Logged in from keytab as oryx/127.10.198.2@KRBTEST.COM (short username oryx)
I20260925 22:45:03.982903 19923 webserver.cc:460] Webserver started at http://127.10.198.2:39483/ using document root <none> and password file <none>
I20260925 22:45:03.983554 19923 fs_manager.cc:359] Metadata directory not provided
I20260925 22:45:03.983639 19923 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:45:03.988102 19923 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260925 22:45:03.990621 19938 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:45:03.992156 19923 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 22:45:03.992301 19923 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376259650642-11032-0/minicluster-data/ts-1/data,/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376259650642-11032-0/minicluster-data/ts-1/wal
uuid: "5b1a0e3292234ad5bd61eb1e0d834157"
format_stamp: "Formatted at 2026-09-25 22:45:02 on dist-test-slave-5vdr"
server_key: "5f8d7c88d28053c7a95cc459b31c0351"
server_key_iv: "62677311ddd2dc526bfe2a30eef7a1c8"
server_key_version: "encryptionkey@0"
I20260925 22:45:03.992911 19923 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376259650642-11032-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376259650642-11032-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376259650642-11032-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:45:04.012889 19923 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:45:04.016762 19923 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:45:04.016959 19923 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:45:04.017705 19923 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 22:45:04.018694 19923 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:45:04.018774 19923 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:45:04.018848 19923 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:45:04.018890 19923 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 22:45:04.029060 19923 rpc_server.cc:307] RPC server started. Bound to: 127.10.198.2:41367
I20260925 22:45:04.029085 20051 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.198.2:41367 every 8 connection(s)
I20260925 22:45:04.030296 19923 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376259650642-11032-0/minicluster-data/ts-1/data/info.pb
I20260925 22:45:04.038635 11032 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrWXm8X/build/debug/bin/kudu as pid 19923
I20260925 22:45:04.039402 11032 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrWXm8X/build/debug/bin/kudu
/tmp/dist-test-taskrWXm8X/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376259650642-11032-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376259650642-11032-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376259650642-11032-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376259650642-11032-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.198.3:42213
--local_ip_for_outbound_sockets=127.10.198.3
--tserver_master_addrs=127.10.198.62:32879
--webserver_port=42697
--webserver_interface=127.10.198.3
--keytab_file=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376259650642-11032-0/krb5kdc/kudu_127.10.198.3.keytab
--principal=kudu/127.10.198.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.198.20:33781
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.10.198.3
--keytab_file=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376259650642-11032-0/krb5kdc/oryx_127.10.198.3.keytab with env {KRB5CCNAME=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376259650642-11032-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376259650642-11032-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376259650642-11032-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 25 22:45:04 dist-test-slave-5vdr krb5kdc[19188](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.198.2@KRBTEST.COM for oryx/127.10.198.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 22:45:04 dist-test-slave-5vdr krb5kdc[19188](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.198.2@KRBTEST.COM for oryx/127.10.198.62@KRBTEST.COM, Server not found in Kerberos database
W20260925 22:45:04.044528 19921 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:45:04.032814 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.198.2:40529 (local address 127.10.198.62:32879)
0925 22:45:04.033048 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0925 22:45:04.033053 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:45:04.034162 (+ 1109us) server_negotiation.cc:407] Connection header received
0925 22:45:04.035572 (+ 1410us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:45:04.035575 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:45:04.035642 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:45:04.035735 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:45:04.037459 (+ 1724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:45:04.038142 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:45:04.039837 (+ 1695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:45:04.040151 (+ 314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:45:04.040224 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:45:04.044395 (+ 4171us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.198.2:40529: BlockingRecv error: recv got EOF from 127.10.198.2:40529 (error 108)
Metrics: {"server-negotiator.queue_time_us":126}
I20260925 22:45:04.045279 20054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:45:04.033366 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.198.62:32879 (local address 127.10.198.2:40529)
0925 22:45:04.033952 (+ 586us) negotiation.cc:107] Waiting for socket to connect
0925 22:45:04.034003 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0925 22:45:04.035262 (+ 1259us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:45:04.036044 (+ 782us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:45:04.036054 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:45:04.036492 (+ 438us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:45:04.037216 (+ 724us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:45:04.037235 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:45:04.038315 (+ 1080us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:45:04.038332 (+ 17us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:45:04.039604 (+ 1272us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:45:04.039621 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:45:04.040454 (+ 833us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:45:04.040457 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:45:04.040636 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:45:04.041451 (+ 815us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:45:04.041471 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:45:04.044246 (+ 2775us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.198.62:32879: Server oryx/127.10.198.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":356,"thread_start_us":154,"threads_started":1}
W20260925 22:45:04.045866 20052 heartbeater.cc:644] Failed to heartbeat to 127.10.198.62:32879 (0 consecutive failures): Not authorized: Failed to ping master at 127.10.198.62:32879: Client connection negotiation failed: client connection to 127.10.198.62:32879: Server oryx/127.10.198.62@KRBTEST.COM not found in Kerberos database
Sep 25 22:45:04 dist-test-slave-5vdr krb5kdc[19188](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.198.2@KRBTEST.COM for oryx/127.10.198.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 22:45:04 dist-test-slave-5vdr krb5kdc[19188](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.198.2@KRBTEST.COM for oryx/127.10.198.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 22:45:04.052179 20054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:45:04.046494 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.198.62:32879 (local address 127.10.198.2:40581)
0925 22:45:04.046637 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0925 22:45:04.046655 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0925 22:45:04.046761 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:45:04.047060 (+ 299us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:45:04.047063 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:45:04.047245 (+ 182us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:45:04.047345 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:45:04.047350 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:45:04.048189 (+ 839us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:45:04.048193 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:45:04.048765 (+ 572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:45:04.048773 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:45:04.049297 (+ 524us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:45:04.049301 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:45:04.049406 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:45:04.049959 (+ 553us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:45:04.049972 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:45:04.051993 (+ 2021us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.198.62:32879: Server oryx/127.10.198.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":54}
W20260925 22:45:04.052266 19921 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:45:04.046600 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.198.2:40581 (local address 127.10.198.62:32879)
0925 22:45:04.046726 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0925 22:45:04.046730 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:45:04.046744 (+ 14us) server_negotiation.cc:407] Connection header received
0925 22:45:04.046898 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:45:04.046901 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:45:04.046944 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:45:04.047031 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:45:04.047464 (+ 433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:45:04.048072 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:45:04.048936 (+ 864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:45:04.049169 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:45:04.049290 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:45:04.052145 (+ 2855us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.198.2:40581: BlockingRecv error: recv got EOF from 127.10.198.2:40581 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
Sep 25 22:45:04 dist-test-slave-5vdr krb5kdc[19188](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.198.2@KRBTEST.COM for oryx/127.10.198.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 22:45:04 dist-test-slave-5vdr krb5kdc[19188](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.198.2@KRBTEST.COM for oryx/127.10.198.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 22:45:04.059662 20054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:45:04.053108 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.198.62:32879 (local address 127.10.198.2:53485)
0925 22:45:04.053325 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0925 22:45:04.053343 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0925 22:45:04.053451 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:45:04.053746 (+ 295us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:45:04.053749 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:45:04.053961 (+ 212us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:45:04.054085 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:45:04.054093 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:45:04.054872 (+ 779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:45:04.054876 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:45:04.055824 (+ 948us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:45:04.055869 (+ 45us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:45:04.056378 (+ 509us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:45:04.056383 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:45:04.056484 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:45:04.057032 (+ 548us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:45:04.057051 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:45:04.059488 (+ 2437us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.198.62:32879: Server oryx/127.10.198.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":127}
W20260925 22:45:04.059957 20052 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260925 22:45:04.060109 19921 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:45:04.053195 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.198.2:53485 (local address 127.10.198.62:32879)
0925 22:45:04.053321 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0925 22:45:04.053324 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 22:45:04.053410 (+ 86us) server_negotiation.cc:407] Connection header received
0925 22:45:04.053550 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:45:04.053553 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:45:04.053598 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:45:04.053671 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:45:04.054228 (+ 557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:45:04.054749 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:45:04.056002 (+ 1253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:45:04.056236 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:45:04.056329 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:45:04.059666 (+ 3337us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.198.2:53485: BlockingRecv error: recv got EOF from 127.10.198.2:53485 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
W20260925 22:45:04.169414 20055 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:45:04.169734 20055 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:45:04.169806 20055 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:45:04.174273 20055 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:45:04.174360 20055 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:45:04.174470 20055 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.198.3
I20260925 22:45:04.180812 20055 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.198.20:33781
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376259650642-11032-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376259650642-11032-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376259650642-11032-0/krb5kdc/oryx_127.10.198.3.keytab
--principal=oryx/127.10.198.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.198.3:42213
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376259650642-11032-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.198.3
--webserver_port=42697
--tserver_master_addrs=127.10.198.62:32879
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20055
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.198.3
--log_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376259650642-11032-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:09:16 UTC on ccycloud.quasar-afmhyw.root.comops.site
build id 4673379
I20260925 22:45:04.182058 20055 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:45:04.183182 20055 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:45:04.194726 20063 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:45:04.194726 20061 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:45:04.194727 20060 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:45:04.195262 20055 server_base.cc:1034] running on GCE node
I20260925 22:45:04.195756 20055 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:45:04.197121 20055 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:45:04.198371 20055 hybrid_clock.cc:648] HybridClock initialized: now 1790376304198303 us; error 85 us; skew 500 ppm
Sep 25 22:45:04 dist-test-slave-5vdr krb5kdc[19188](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376304, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.198.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:45:04.201763 20055 init.cc:377] Logged in from keytab as oryx/127.10.198.3@KRBTEST.COM (short username oryx)
I20260925 22:45:04.202888 20055 webserver.cc:460] Webserver started at http://127.10.198.3:42697/ using document root <none> and password file <none>
I20260925 22:45:04.203497 20055 fs_manager.cc:359] Metadata directory not provided
I20260925 22:45:04.203584 20055 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:45:04.208287 20055 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260925 22:45:04.210786 20070 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:45:04.212448 20055 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 22:45:04.212646 20055 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376259650642-11032-0/minicluster-data/ts-2/data,/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376259650642-11032-0/minicluster-data/ts-2/wal
uuid: "fe69c4947a5b4d6eabf4749c51867ffb"
format_stamp: "Formatted at 2026-09-25 22:45:03 on dist-test-slave-5vdr"
server_key: "8fd08cea63366d85477442881b0c7893"
server_key_iv: "e507616b1b02b3dbdf7ec14dca6945b2"
server_key_version: "encryptionkey@0"
I20260925 22:45:04.213294 20055 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376259650642-11032-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376259650642-11032-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376259650642-11032-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:45:04.239514 20055 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:45:04.243235 20055 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:45:04.243464 20055 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:45:04.244143 20055 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 22:45:04.245059 20055 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:45:04.245110 20055 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:45:04.245179 20055 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:45:04.245213 20055 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 22:45:04.255227 20055 rpc_server.cc:307] RPC server started. Bound to: 127.10.198.3:42213
I20260925 22:45:04.255294 20183 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.198.3:42213 every 8 connection(s)
I20260925 22:45:04.256584 20055 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790376259650642-11032-0/minicluster-data/ts-2/data/info.pb
I20260925 22:45:04.258129 11032 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrWXm8X/build/debug/bin/kudu as pid 20055
Sep 25 22:45:04 dist-test-slave-5vdr krb5kdc[19188](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.198.3@KRBTEST.COM for oryx/127.10.198.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 22:45:04 dist-test-slave-5vdr krb5kdc[19188](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.198.3@KRBTEST.COM for oryx/127.10.198.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 22:45:04.269742 20187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:45:04.259297 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47828 (local address 127.10.198.62:32879)
0925 22:45:04.259527 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0925 22:45:04.259530 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 22:45:04.259543 (+ 13us) server_negotiation.cc:407] Connection header received
0925 22:45:04.259586 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:45:04.259606 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:45:04.259664 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:45:04.259771 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:45:04.260720 (+ 949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:45:04.261556 (+ 836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:45:04.262569 (+ 1013us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:45:04.262905 (+ 336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:45:04.263032 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:45:04.264763 (+ 1731us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:45:04.264770 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:45:04.264793 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:45:04.264835 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:45:04.267157 (+ 2322us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:45:04.267698 (+ 541us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:45:04.267708 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:45:04.267719 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:45:04.267981 (+ 262us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:45:04.268322 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:45:04.268324 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:45:04.268325 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:45:04.268779 (+ 454us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:45:04.268951 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0925 22:45:04.269309 (+ 358us) server_negotiation.cc:299] Negotiation successful
0925 22:45:04.269605 (+ 296us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":167,"thread_start_us":100,"threads_started":1}
W20260925 22:45:04.270658 19921 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:45:04.258778 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.198.3:49625 (local address 127.10.198.62:32879)
0925 22:45:04.258921 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0925 22:45:04.258926 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:45:04.260132 (+ 1206us) server_negotiation.cc:407] Connection header received
0925 22:45:04.261754 (+ 1622us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:45:04.261757 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:45:04.261810 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:45:04.261879 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:45:04.263488 (+ 1609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:45:04.264085 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:45:04.265718 (+ 1633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:45:04.266088 (+ 370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:45:04.266188 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:45:04.270550 (+ 4362us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.198.3:49625: BlockingRecv error: recv got EOF from 127.10.198.3:49625 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
I20260925 22:45:04.271432 20188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:45:04.259317 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.198.62:32879 (local address 127.10.198.3:49625)
0925 22:45:04.259925 (+ 608us) negotiation.cc:107] Waiting for socket to connect
0925 22:45:04.259983 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0925 22:45:04.261528 (+ 1545us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:45:04.262095 (+ 567us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:45:04.262103 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:45:04.262614 (+ 511us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:45:04.263323 (+ 709us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:45:04.263342 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:45:04.264219 (+ 877us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:45:04.264229 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:45:04.265551 (+ 1322us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:45:04.265569 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:45:04.266272 (+ 703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:45:04.266277 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:45:04.266518 (+ 241us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:45:04.267095 (+ 577us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:45:04.267119 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:45:04.270445 (+ 3326us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.198.62:32879: Server oryx/127.10.198.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":328,"thread_start_us":143,"threads_started":1}
W20260925 22:45:04.272089 20184 heartbeater.cc:644] Failed to heartbeat to 127.10.198.62:32879 (0 consecutive failures): Not authorized: Failed to ping master at 127.10.198.62:32879: Client connection negotiation failed: client connection to 127.10.198.62:32879: Server oryx/127.10.198.62@KRBTEST.COM not found in Kerberos database
Sep 25 22:45:04 dist-test-slave-5vdr krb5kdc[19188](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.198.3@KRBTEST.COM for oryx/127.10.198.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 22:45:04 dist-test-slave-5vdr krb5kdc[19188](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.198.3@KRBTEST.COM for oryx/127.10.198.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 22:45:04.277891 20188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:45:04.272635 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.198.62:32879 (local address 127.10.198.3:41961)
0925 22:45:04.272750 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0925 22:45:04.272767 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 22:45:04.272852 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:45:04.273133 (+ 281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:45:04.273135 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:45:04.273301 (+ 166us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:45:04.273409 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:45:04.273415 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:45:04.274172 (+ 757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:45:04.274174 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:45:04.274734 (+ 560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:45:04.274743 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:45:04.275172 (+ 429us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:45:04.275174 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:45:04.275248 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:45:04.275827 (+ 579us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:45:04.275891 (+ 64us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:45:04.277773 (+ 1882us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.198.62:32879: Server oryx/127.10.198.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":33}
W20260925 22:45:04.278074 19921 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:45:04.272699 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.198.3:41961 (local address 127.10.198.62:32879)
0925 22:45:04.272825 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0925 22:45:04.272832 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 22:45:04.272848 (+ 16us) server_negotiation.cc:407] Connection header received
0925 22:45:04.272943 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:45:04.272946 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:45:04.273004 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:45:04.273084 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:45:04.273539 (+ 455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:45:04.274066 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:45:04.274845 (+ 779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:45:04.275059 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:45:04.275139 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:45:04.277937 (+ 2798us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.198.3:41961: BlockingRecv error: recv got EOF from 127.10.198.3:41961 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
Sep 25 22:45:04 dist-test-slave-5vdr krb5kdc[19188](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.198.3@KRBTEST.COM for oryx/127.10.198.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 22:45:04 dist-test-slave-5vdr krb5kdc[19188](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.198.3@KRBTEST.COM for oryx/127.10.198.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 22:45:04.284162 20188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:45:04.278526 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.198.62:32879 (local address 127.10.198.3:47119)
0925 22:45:04.278666 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0925 22:45:04.278683 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 22:45:04.278766 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:45:04.278968 (+ 202us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:45:04.278970 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:45:04.279113 (+ 143us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:45:04.279199 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:45:04.279205 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:45:04.280036 (+ 831us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:45:04.280039 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:45:04.280595 (+ 556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:45:04.280606 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:45:04.281145 (+ 539us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:45:04.281149 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:45:04.281300 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:45:04.281883 (+ 583us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:45:04.281896 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:45:04.283991 (+ 2095us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.198.62:32879: Server oryx/127.10.198.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":59}
W20260925 22:45:04.284446 20184 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260925 22:45:04.284246 19921 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:45:04.278583 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.198.3:47119 (local address 127.10.198.62:32879)
0925 22:45:04.278761 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0925 22:45:04.278764 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 22:45:04.278779 (+ 15us) server_negotiation.cc:407] Connection header received
0925 22:45:04.278824 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:45:04.278826 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:45:04.278864 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:45:04.278937 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:45:04.279304 (+ 367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:45:04.279905 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:45:04.280746 (+ 841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:45:04.281003 (+ 257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:45:04.281102 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:45:04.284121 (+ 3019us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.198.3:47119: BlockingRecv error: recv got EOF from 127.10.198.3:47119 (error 108)
Metrics: {"server-negotiator.queue_time_us":98}
Sep 25 22:45:04 dist-test-slave-5vdr krb5kdc[19188](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.198.1@KRBTEST.COM for oryx/127.10.198.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 22:45:04 dist-test-slave-5vdr krb5kdc[19188](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.198.1@KRBTEST.COM for oryx/127.10.198.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 22:45:04.860673 20189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:45:04.847800 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.198.62:32879 (local address 127.10.198.1:45331)
0925 22:45:04.848201 (+ 401us) negotiation.cc:107] Waiting for socket to connect
0925 22:45:04.848218 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 22:45:04.848353 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:45:04.848842 (+ 489us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:45:04.848847 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:45:04.849128 (+ 281us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:45:04.849322 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:45:04.849330 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:45:04.850265 (+ 935us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:45:04.850268 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:45:04.851072 (+ 804us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:45:04.851086 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:45:04.851741 (+ 655us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:45:04.851755 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:45:04.851934 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:45:04.858039 (+ 6105us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:45:04.858057 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:45:04.860517 (+ 2460us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.198.62:32879: Server oryx/127.10.198.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":294,"thread_start_us":103,"threads_started":1}
W20260925 22:45:04.860929 20190 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:45:04.847900 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.198.1:45331 (local address 127.10.198.62:32879)
0925 22:45:04.848179 (+ 279us) server_negotiation.cc:206] Beginning negotiation
0925 22:45:04.848186 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 22:45:04.848405 (+ 219us) server_negotiation.cc:407] Connection header received
0925 22:45:04.848499 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:45:04.848503 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:45:04.848590 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:45:04.848751 (+ 161us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:45:04.849502 (+ 751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:45:04.850069 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:45:04.851247 (+ 1178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:45:04.851516 (+ 269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:45:04.851626 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:45:04.860763 (+ 9137us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.198.1:45331: BlockingRecv error: recv got EOF from 127.10.198.1:45331 (error 108)
Metrics: {"server-negotiator.queue_time_us":179,"thread_start_us":63,"threads_started":1}
Sep 25 22:45:05 dist-test-slave-5vdr krb5kdc[19188](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.198.2@KRBTEST.COM for oryx/127.10.198.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 22:45:05 dist-test-slave-5vdr krb5kdc[19188](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.198.2@KRBTEST.COM for oryx/127.10.198.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 22:45:05.067373 20191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:45:05.060938 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.198.62:32879 (local address 127.10.198.2:46983)
0925 22:45:05.061236 (+ 298us) negotiation.cc:107] Waiting for socket to connect
0925 22:45:05.061251 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 22:45:05.061508 (+ 257us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:45:05.061959 (+ 451us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:45:05.061963 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:45:05.062205 (+ 242us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:45:05.062339 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:45:05.062348 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:45:05.063397 (+ 1049us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:45:05.063400 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:45:05.064042 (+ 642us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:45:05.064052 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:45:05.064499 (+ 447us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:45:05.064502 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:45:05.064566 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:45:05.065188 (+ 622us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:45:05.065207 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:45:05.067235 (+ 2028us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.198.62:32879: Server oryx/127.10.198.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":222,"thread_start_us":105,"threads_started":1}
W20260925 22:45:05.067621 20190 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:45:05.061029 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.198.2:46983 (local address 127.10.198.62:32879)
0925 22:45:05.061192 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0925 22:45:05.061199 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 22:45:05.061346 (+ 147us) server_negotiation.cc:407] Connection header received
0925 22:45:05.061714 (+ 368us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:45:05.061718 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:45:05.061780 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:45:05.061878 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:45:05.062512 (+ 634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:45:05.063274 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:45:05.064150 (+ 876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:45:05.064369 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:45:05.064464 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:45:05.067497 (+ 3033us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.198.2:46983: BlockingRecv error: recv got EOF from 127.10.198.2:46983 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
Sep 25 22:45:05 dist-test-slave-5vdr krb5kdc[19188](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.198.3@KRBTEST.COM for oryx/127.10.198.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 22:45:05 dist-test-slave-5vdr krb5kdc[19188](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.198.3@KRBTEST.COM for oryx/127.10.198.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 22:45:05.291703 20192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:45:05.285208 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.198.62:32879 (local address 127.10.198.3:57383)
0925 22:45:05.285582 (+ 374us) negotiation.cc:107] Waiting for socket to connect
0925 22:45:05.285601 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0925 22:45:05.285770 (+ 169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:45:05.286124 (+ 354us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:45:05.286127 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:45:05.286378 (+ 251us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:45:05.286517 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:45:05.286536 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:45:05.287478 (+ 942us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:45:05.287484 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:45:05.288118 (+ 634us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:45:05.288130 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:45:05.288661 (+ 531us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:45:05.288665 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:45:05.288765 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:45:05.289422 (+ 657us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:45:05.289445 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:45:05.291559 (+ 2114us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.198.62:32879: Server oryx/127.10.198.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":264,"thread_start_us":91,"threads_started":1}
W20260925 22:45:05.291895 20190 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:45:05.285335 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.198.3:57383 (local address 127.10.198.62:32879)
0925 22:45:05.285554 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0925 22:45:05.285560 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 22:45:05.285710 (+ 150us) server_negotiation.cc:407] Connection header received
0925 22:45:05.285921 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:45:05.285924 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:45:05.285978 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:45:05.286093 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:45:05.286665 (+ 572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:45:05.287336 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:45:05.288267 (+ 931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:45:05.288516 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:45:05.288643 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:45:05.291742 (+ 3099us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.198.3:57383: BlockingRecv error: recv got EOF from 127.10.198.3:57383 (error 108)
Metrics: {"server-negotiator.queue_time_us":106}
Sep 25 22:45:05 dist-test-slave-5vdr krb5kdc[19188](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.198.1@KRBTEST.COM for oryx/127.10.198.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 22:45:05 dist-test-slave-5vdr krb5kdc[19188](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.198.1@KRBTEST.COM for oryx/127.10.198.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 22:45:05.867645 20193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:45:05.861666 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.198.62:32879 (local address 127.10.198.1:40381)
0925 22:45:05.861949 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0925 22:45:05.861971 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0925 22:45:05.862068 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:45:05.862310 (+ 242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:45:05.862312 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:45:05.862496 (+ 184us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:45:05.862599 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:45:05.862604 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:45:05.863505 (+ 901us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:45:05.863510 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:45:05.864111 (+ 601us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:45:05.864121 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:45:05.864657 (+ 536us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:45:05.864660 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:45:05.864747 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:45:05.865347 (+ 600us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:45:05.865361 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:45:05.867496 (+ 2135us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.198.62:32879: Server oryx/127.10.198.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":104,"threads_started":1}
W20260925 22:45:05.867923 20194 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:45:05.861850 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.198.1:40381 (local address 127.10.198.62:32879)
0925 22:45:05.862082 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0925 22:45:05.862087 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:45:05.862104 (+ 17us) server_negotiation.cc:407] Connection header received
0925 22:45:05.862148 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:45:05.862150 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:45:05.862202 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:45:05.862296 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:45:05.862753 (+ 457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:45:05.863367 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:45:05.864258 (+ 891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:45:05.864511 (+ 253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:45:05.864636 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:45:05.867704 (+ 3068us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.198.1:40381: BlockingRecv error: recv got EOF from 127.10.198.1:40381 (error 108)
Metrics: {"server-negotiator.queue_time_us":156,"thread_start_us":91,"threads_started":1}
Sep 25 22:45:06 dist-test-slave-5vdr krb5kdc[19188](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.198.2@KRBTEST.COM for oryx/127.10.198.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 22:45:06 dist-test-slave-5vdr krb5kdc[19188](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.198.2@KRBTEST.COM for oryx/127.10.198.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 22:45:06.074470 20195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:45:06.068387 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.198.62:32879 (local address 127.10.198.2:60927)
0925 22:45:06.068701 (+ 314us) negotiation.cc:107] Waiting for socket to connect
0925 22:45:06.068716 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 22:45:06.068818 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:45:06.069116 (+ 298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:45:06.069119 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:45:06.069369 (+ 250us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:45:06.069478 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:45:06.069484 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:45:06.070253 (+ 769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:45:06.070256 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:45:06.070868 (+ 612us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:45:06.070878 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:45:06.071399 (+ 521us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:45:06.071402 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:45:06.071480 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:45:06.072240 (+ 760us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:45:06.072255 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:45:06.074312 (+ 2057us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.198.62:32879: Server oryx/127.10.198.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":233,"thread_start_us":103,"threads_started":1}
W20260925 22:45:06.074581 20194 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:45:06.068482 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.198.2:60927 (local address 127.10.198.62:32879)
0925 22:45:06.068640 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0925 22:45:06.068645 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:45:06.068798 (+ 153us) server_negotiation.cc:407] Connection header received
0925 22:45:06.068912 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:45:06.068915 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:45:06.068969 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:45:06.069051 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:45:06.069588 (+ 537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:45:06.070120 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:45:06.070987 (+ 867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:45:06.071225 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:45:06.071313 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:45:06.074479 (+ 3166us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.198.2:60927: BlockingRecv error: recv got EOF from 127.10.198.2:60927 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
Sep 25 22:45:06 dist-test-slave-5vdr krb5kdc[19188](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.198.3@KRBTEST.COM for oryx/127.10.198.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 22:45:06 dist-test-slave-5vdr krb5kdc[19188](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.198.3@KRBTEST.COM for oryx/127.10.198.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 22:45:06.298755 20196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:45:06.292647 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.198.62:32879 (local address 127.10.198.3:43353)
0925 22:45:06.292964 (+ 317us) negotiation.cc:107] Waiting for socket to connect
0925 22:45:06.292979 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 22:45:06.293074 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:45:06.293336 (+ 262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:45:06.293338 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:45:06.293520 (+ 182us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:45:06.293640 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:45:06.293646 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:45:06.294442 (+ 796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:45:06.294445 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:45:06.295046 (+ 601us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:45:06.295057 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:45:06.295628 (+ 571us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:45:06.295632 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:45:06.295700 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:45:06.296328 (+ 628us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:45:06.296345 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:45:06.298601 (+ 2256us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.198.62:32879: Server oryx/127.10.198.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":214,"thread_start_us":121,"threads_started":1}
W20260925 22:45:06.298992 20194 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:45:06.292861 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.198.3:43353 (local address 127.10.198.62:32879)
0925 22:45:06.293009 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0925 22:45:06.293013 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:45:06.293029 (+ 16us) server_negotiation.cc:407] Connection header received
0925 22:45:06.293156 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:45:06.293158 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:45:06.293206 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:45:06.293295 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:45:06.293769 (+ 474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:45:06.294309 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:45:06.295186 (+ 877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:45:06.295472 (+ 286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:45:06.295586 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:45:06.298828 (+ 3242us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.198.3:43353: BlockingRecv error: recv got EOF from 127.10.198.3:43353 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
Sep 25 22:45:06 dist-test-slave-5vdr krb5kdc[19188](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.198.1@KRBTEST.COM for oryx/127.10.198.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 22:45:06 dist-test-slave-5vdr krb5kdc[19188](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.198.1@KRBTEST.COM for oryx/127.10.198.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 22:45:06.874861 20197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:45:06.868886 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.198.62:32879 (local address 127.10.198.1:48479)
0925 22:45:06.869204 (+ 318us) negotiation.cc:107] Waiting for socket to connect
0925 22:45:06.869222 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0925 22:45:06.869339 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:45:06.869640 (+ 301us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:45:06.869642 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:45:06.869844 (+ 202us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:45:06.869957 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:45:06.869964 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:45:06.870754 (+ 790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:45:06.870757 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:45:06.871371 (+ 614us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:45:06.871381 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:45:06.871973 (+ 592us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:45:06.871976 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:45:06.872050 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:45:06.872691 (+ 641us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:45:06.872705 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:45:06.874712 (+ 2007us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.198.62:32879: Server oryx/127.10.198.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":225,"thread_start_us":134,"threads_started":1}
W20260925 22:45:06.875133 20198 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:45:06.868997 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.198.1:48479 (local address 127.10.198.62:32879)
0925 22:45:06.869260 (+ 263us) server_negotiation.cc:206] Beginning negotiation
0925 22:45:06.869264 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:45:06.869282 (+ 18us) server_negotiation.cc:407] Connection header received
0925 22:45:06.869437 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:45:06.869439 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:45:06.869485 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:45:06.869577 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:45:06.870077 (+ 500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:45:06.870612 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:45:06.871512 (+ 900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:45:06.871743 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:45:06.871830 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:45:06.874980 (+ 3150us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.198.1:48479: BlockingRecv error: recv got EOF from 127.10.198.1:48479 (error 108)
Metrics: {"server-negotiator.queue_time_us":179,"thread_start_us":107,"threads_started":1}
Sep 25 22:45:07 dist-test-slave-5vdr krb5kdc[19188](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.198.2@KRBTEST.COM for oryx/127.10.198.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 22:45:07 dist-test-slave-5vdr krb5kdc[19188](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.198.2@KRBTEST.COM for oryx/127.10.198.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 22:45:07.081846 20199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:45:07.075770 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.198.62:32879 (local address 127.10.198.2:58495)
0925 22:45:07.076093 (+ 323us) negotiation.cc:107] Waiting for socket to connect
0925 22:45:07.076107 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 22:45:07.076212 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:45:07.076487 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:45:07.076490 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:45:07.076675 (+ 185us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:45:07.076777 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:45:07.076783 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:45:07.077619 (+ 836us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:45:07.077623 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:45:07.078201 (+ 578us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:45:07.078211 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:45:07.078802 (+ 591us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:45:07.078805 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:45:07.078878 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:45:07.079421 (+ 543us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:45:07.079433 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:45:07.081665 (+ 2232us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.198.62:32879: Server oryx/127.10.198.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":236,"thread_start_us":95,"threads_started":1}
W20260925 22:45:07.081964 20198 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:45:07.075943 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.198.2:58495 (local address 127.10.198.62:32879)
0925 22:45:07.076092 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0925 22:45:07.076096 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:45:07.076180 (+ 84us) server_negotiation.cc:407] Connection header received
0925 22:45:07.076297 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:45:07.076299 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:45:07.076352 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:45:07.076432 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:45:07.076918 (+ 486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:45:07.077469 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:45:07.078375 (+ 906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:45:07.078638 (+ 263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:45:07.078736 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:45:07.081828 (+ 3092us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.198.2:58495: BlockingRecv error: recv got EOF from 127.10.198.2:58495 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Sep 25 22:45:07 dist-test-slave-5vdr krb5kdc[19188](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.198.3@KRBTEST.COM for oryx/127.10.198.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 22:45:07 dist-test-slave-5vdr krb5kdc[19188](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.198.3@KRBTEST.COM for oryx/127.10.198.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 22:45:07.306250 20200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:45:07.299648 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.198.62:32879 (local address 127.10.198.3:37375)
0925 22:45:07.299972 (+ 324us) negotiation.cc:107] Waiting for socket to connect
0925 22:45:07.299986 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 22:45:07.300098 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:45:07.300365 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:45:07.300367 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:45:07.300564 (+ 197us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:45:07.300666 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:45:07.300672 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:45:07.301439 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:45:07.301441 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:45:07.302036 (+ 595us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:45:07.302045 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:45:07.302627 (+ 582us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:45:07.302632 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:45:07.302748 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:45:07.303520 (+ 772us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:45:07.303537 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:45:07.306114 (+ 2577us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.198.62:32879: Server oryx/127.10.198.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":233,"thread_start_us":105,"threads_started":1}
W20260925 22:45:07.306574 20198 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:45:07.299780 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.198.3:37375 (local address 127.10.198.62:32879)
0925 22:45:07.299955 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0925 22:45:07.299959 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:45:07.300062 (+ 103us) server_negotiation.cc:407] Connection header received
0925 22:45:07.300185 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:45:07.300187 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:45:07.300240 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:45:07.300326 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:45:07.300787 (+ 461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:45:07.301331 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:45:07.302161 (+ 830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:45:07.302395 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:45:07.302482 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:45:07.306445 (+ 3963us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.198.3:37375: BlockingRecv error: recv got EOF from 127.10.198.3:37375 (error 108)
Metrics: {"server-negotiator.queue_time_us":78}
Sep 25 22:45:07 dist-test-slave-5vdr krb5kdc[19188](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.198.1@KRBTEST.COM for oryx/127.10.198.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 22:45:07 dist-test-slave-5vdr krb5kdc[19188](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.198.1@KRBTEST.COM for oryx/127.10.198.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 22:45:07.882104 20201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:45:07.875832 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.198.62:32879 (local address 127.10.198.1:55379)
0925 22:45:07.876197 (+ 365us) negotiation.cc:107] Waiting for socket to connect
0925 22:45:07.876212 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 22:45:07.876310 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:45:07.876543 (+ 233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:45:07.876545 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:45:07.876733 (+ 188us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:45:07.876837 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:45:07.876843 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:45:07.877715 (+ 872us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:45:07.877718 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:45:07.878377 (+ 659us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:45:07.878393 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:45:07.878902 (+ 509us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:45:07.878905 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:45:07.878977 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:45:07.879589 (+ 612us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:45:07.879602 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:45:07.881975 (+ 2373us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.198.62:32879: Server oryx/127.10.198.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":235,"thread_start_us":103,"threads_started":1}
W20260925 22:45:07.882400 20202 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:45:07.876035 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.198.1:55379 (local address 127.10.198.62:32879)
0925 22:45:07.876310 (+ 275us) server_negotiation.cc:206] Beginning negotiation
0925 22:45:07.876313 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 22:45:07.876328 (+ 15us) server_negotiation.cc:407] Connection header received
0925 22:45:07.876374 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:45:07.876376 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:45:07.876419 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:45:07.876502 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:45:07.876993 (+ 491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:45:07.877595 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:45:07.878511 (+ 916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:45:07.878758 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:45:07.878847 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:45:07.882220 (+ 3373us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.198.1:55379: BlockingRecv error: recv got EOF from 127.10.198.1:55379 (error 108)
Metrics: {"server-negotiator.queue_time_us":178,"thread_start_us":99,"threads_started":1}
Sep 25 22:45:08 dist-test-slave-5vdr krb5kdc[19188](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.198.2@KRBTEST.COM for oryx/127.10.198.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 22:45:08 dist-test-slave-5vdr krb5kdc[19188](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.198.2@KRBTEST.COM for oryx/127.10.198.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 22:45:08.089238 20203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:45:08.082947 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.198.62:32879 (local address 127.10.198.2:35915)
0925 22:45:08.083328 (+ 381us) negotiation.cc:107] Waiting for socket to connect
0925 22:45:08.083344 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 22:45:08.083524 (+ 180us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:45:08.083889 (+ 365us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:45:08.083892 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:45:08.084133 (+ 241us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:45:08.084226 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:45:08.084232 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:45:08.085024 (+ 792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:45:08.085029 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:45:08.085604 (+ 575us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:45:08.085613 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:45:08.086119 (+ 506us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:45:08.086123 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:45:08.086215 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:45:08.086822 (+ 607us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:45:08.086834 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:45:08.089102 (+ 2268us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.198.62:32879: Server oryx/127.10.198.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":308,"thread_start_us":136,"threads_started":1}
W20260925 22:45:08.089511 20202 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:45:08.083131 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.198.2:35915 (local address 127.10.198.62:32879)
0925 22:45:08.083314 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0925 22:45:08.083319 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:45:08.083438 (+ 119us) server_negotiation.cc:407] Connection header received
0925 22:45:08.083662 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:45:08.083665 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:45:08.083717 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:45:08.083871 (+ 154us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:45:08.084350 (+ 479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:45:08.084888 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:45:08.085736 (+ 848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:45:08.085981 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:45:08.086097 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:45:08.089378 (+ 3281us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.198.2:35915: BlockingRecv error: recv got EOF from 127.10.198.2:35915 (error 108)
Metrics: {"server-negotiator.queue_time_us":80}
Sep 25 22:45:08 dist-test-slave-5vdr krb5kdc[19188](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.198.3@KRBTEST.COM for oryx/127.10.198.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 22:45:08 dist-test-slave-5vdr krb5kdc[19188](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.198.3@KRBTEST.COM for oryx/127.10.198.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 22:45:08.313954 20204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:45:08.307654 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.198.62:32879 (local address 127.10.198.3:54089)
0925 22:45:08.308128 (+ 474us) negotiation.cc:107] Waiting for socket to connect
0925 22:45:08.308151 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0925 22:45:08.308305 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:45:08.308610 (+ 305us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:45:08.308613 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:45:08.308837 (+ 224us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:45:08.308935 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:45:08.308941 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:45:08.309727 (+ 786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:45:08.309732 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:45:08.310303 (+ 571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:45:08.310312 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:45:08.310800 (+ 488us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:45:08.310803 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:45:08.310885 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:45:08.311618 (+ 733us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:45:08.311632 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:45:08.313768 (+ 2136us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.198.62:32879: Server oryx/127.10.198.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":395,"thread_start_us":96,"threads_started":1}
W20260925 22:45:08.314042 20202 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:45:08.307815 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.198.3:54089 (local address 127.10.198.62:32879)
0925 22:45:08.308095 (+ 280us) server_negotiation.cc:206] Beginning negotiation
0925 22:45:08.308100 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:45:08.308239 (+ 139us) server_negotiation.cc:407] Connection header received
0925 22:45:08.308421 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:45:08.308424 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:45:08.308489 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:45:08.308586 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:45:08.309065 (+ 479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:45:08.309603 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:45:08.310439 (+ 836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:45:08.310674 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:45:08.310792 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:45:08.313925 (+ 3133us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.198.3:54089: BlockingRecv error: recv got EOF from 127.10.198.3:54089 (error 108)
Metrics: {"server-negotiator.queue_time_us":177}
Sep 25 22:45:08 dist-test-slave-5vdr krb5kdc[19188](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.198.1@KRBTEST.COM for oryx/127.10.198.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 22:45:08 dist-test-slave-5vdr krb5kdc[19188](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.198.1@KRBTEST.COM for oryx/127.10.198.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 22:45:08.889334 20205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:45:08.883064 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.198.62:32879 (local address 127.10.198.1:53625)
0925 22:45:08.883440 (+ 376us) negotiation.cc:107] Waiting for socket to connect
0925 22:45:08.883454 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 22:45:08.883550 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:45:08.883916 (+ 366us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:45:08.883923 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:45:08.884127 (+ 204us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:45:08.884228 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:45:08.884250 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:45:08.885124 (+ 874us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:45:08.885127 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:45:08.885951 (+ 824us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:45:08.885964 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:45:08.886593 (+ 629us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:45:08.886596 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:45:08.886663 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:45:08.887179 (+ 516us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:45:08.887191 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:45:08.889206 (+ 2015us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.198.62:32879: Server oryx/127.10.198.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":292,"thread_start_us":106,"threads_started":1}
W20260925 22:45:08.889570 20206 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:45:08.883217 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.198.1:53625 (local address 127.10.198.62:32879)
0925 22:45:08.883487 (+ 270us) server_negotiation.cc:206] Beginning negotiation
0925 22:45:08.883490 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 22:45:08.883509 (+ 19us) server_negotiation.cc:407] Connection header received
0925 22:45:08.883647 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:45:08.883649 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:45:08.883702 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:45:08.883813 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:45:08.884370 (+ 557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:45:08.884999 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:45:08.886077 (+ 1078us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:45:08.886470 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:45:08.886567 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:45:08.889417 (+ 2850us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.198.1:53625: BlockingRecv error: recv got EOF from 127.10.198.1:53625 (error 108)
Metrics: {"server-negotiator.queue_time_us":186,"thread_start_us":78,"threads_started":1}
Sep 25 22:45:09 dist-test-slave-5vdr krb5kdc[19188](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.198.2@KRBTEST.COM for oryx/127.10.198.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 22:45:09 dist-test-slave-5vdr krb5kdc[19188](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.198.2@KRBTEST.COM for oryx/127.10.198.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 22:45:09.096560 20207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:45:09.090150 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.198.62:32879 (local address 127.10.198.2:41631)
0925 22:45:09.090646 (+ 496us) negotiation.cc:107] Waiting for socket to connect
0925 22:45:09.090659 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 22:45:09.090837 (+ 178us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:45:09.091179 (+ 342us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:45:09.091183 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:45:09.091460 (+ 277us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:45:09.091561 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:45:09.091567 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:45:09.092402 (+ 835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:45:09.092405 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:45:09.093038 (+ 633us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:45:09.093048 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:45:09.093563 (+ 515us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:45:09.093566 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:45:09.093637 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:45:09.094269 (+ 632us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:45:09.094281 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:45:09.096382 (+ 2101us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.198.62:32879: Server oryx/127.10.198.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":429,"thread_start_us":141,"threads_started":1}
W20260925 22:45:09.096800 20206 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:45:09.090288 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.198.2:41631 (local address 127.10.198.62:32879)
0925 22:45:09.090445 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0925 22:45:09.090452 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 22:45:09.090740 (+ 288us) server_negotiation.cc:407] Connection header received
0925 22:45:09.090958 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:45:09.090961 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:45:09.091014 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:45:09.091094 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:45:09.091692 (+ 598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:45:09.092264 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:45:09.093193 (+ 929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:45:09.093430 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:45:09.093521 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:45:09.096654 (+ 3133us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.198.2:41631: BlockingRecv error: recv got EOF from 127.10.198.2:41631 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Sep 25 22:45:09 dist-test-slave-5vdr krb5kdc[19188](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.198.3@KRBTEST.COM for oryx/127.10.198.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 22:45:09 dist-test-slave-5vdr krb5kdc[19188](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.198.3@KRBTEST.COM for oryx/127.10.198.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 22:45:09.321940 20208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:45:09.315145 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.198.62:32879 (local address 127.10.198.3:52155)
0925 22:45:09.315524 (+ 379us) negotiation.cc:107] Waiting for socket to connect
0925 22:45:09.315544 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0925 22:45:09.315713 (+ 169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:45:09.316160 (+ 447us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:45:09.316164 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:45:09.316370 (+ 206us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:45:09.316510 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:45:09.316517 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:45:09.317328 (+ 811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:45:09.317331 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:45:09.317988 (+ 657us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:45:09.318002 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:45:09.318562 (+ 560us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:45:09.318565 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:45:09.318754 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:45:09.319368 (+ 614us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:45:09.319380 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:45:09.321698 (+ 2318us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.198.62:32879: Server oryx/127.10.198.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":253,"thread_start_us":98,"threads_started":1}
W20260925 22:45:09.322012 20206 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:45:09.315355 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.198.3:52155 (local address 127.10.198.62:32879)
0925 22:45:09.315538 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0925 22:45:09.315542 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:45:09.315638 (+ 96us) server_negotiation.cc:407] Connection header received
0925 22:45:09.315894 (+ 256us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:45:09.315896 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:45:09.315951 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:45:09.316052 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:45:09.316661 (+ 609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:45:09.317219 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:45:09.318176 (+ 957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:45:09.318427 (+ 251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:45:09.318557 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:45:09.321883 (+ 3326us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.198.3:52155: BlockingRecv error: recv got EOF from 127.10.198.3:52155 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
Sep 25 22:45:09 dist-test-slave-5vdr krb5kdc[19188](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.198.1@KRBTEST.COM for oryx/127.10.198.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 22:45:09 dist-test-slave-5vdr krb5kdc[19188](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.198.1@KRBTEST.COM for oryx/127.10.198.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 22:45:09.899801 20209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:45:09.890227 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.198.62:32879 (local address 127.10.198.1:56669)
0925 22:45:09.890505 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0925 22:45:09.890519 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 22:45:09.890615 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:45:09.890862 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:45:09.890864 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:45:09.891052 (+ 188us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:45:09.891152 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:45:09.891158 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:45:09.892062 (+ 904us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:45:09.892065 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:45:09.892714 (+ 649us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:45:09.892725 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:45:09.893234 (+ 509us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:45:09.893237 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:45:09.893302 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:45:09.897189 (+ 3887us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:45:09.897208 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:45:09.899628 (+ 2420us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.198.62:32879: Server oryx/127.10.198.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":96,"threads_started":1}
W20260925 22:45:09.899947 20210 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:45:09.890342 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.198.1:56669 (local address 127.10.198.62:32879)
0925 22:45:09.890610 (+ 268us) server_negotiation.cc:206] Beginning negotiation
0925 22:45:09.890614 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:45:09.890630 (+ 16us) server_negotiation.cc:407] Connection header received
0925 22:45:09.890678 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:45:09.890680 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:45:09.890730 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:45:09.890825 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:45:09.891275 (+ 450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:45:09.891905 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:45:09.892851 (+ 946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:45:09.893093 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:45:09.893178 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:45:09.899779 (+ 6601us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.198.1:56669: BlockingRecv error: recv got EOF from 127.10.198.1:56669 (error 108)
Metrics: {"server-negotiator.queue_time_us":191,"thread_start_us":116,"threads_started":1}
Sep 25 22:45:10 dist-test-slave-5vdr krb5kdc[19188](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.198.2@KRBTEST.COM for oryx/127.10.198.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 22:45:10 dist-test-slave-5vdr krb5kdc[19188](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.198.2@KRBTEST.COM for oryx/127.10.198.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 22:45:10.104126 20211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:45:10.097591 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.198.62:32879 (local address 127.10.198.2:56879)
0925 22:45:10.097883 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0925 22:45:10.097898 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 22:45:10.098013 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:45:10.098297 (+ 284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:45:10.098299 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:45:10.098486 (+ 187us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:45:10.098585 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:45:10.098590 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:45:10.099728 (+ 1138us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:45:10.099731 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:45:10.100433 (+ 702us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:45:10.100444 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:45:10.100958 (+ 514us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:45:10.100961 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:45:10.101027 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:45:10.101566 (+ 539us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:45:10.101580 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:45:10.103953 (+ 2373us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.198.62:32879: Server oryx/127.10.198.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":202,"thread_start_us":76,"threads_started":1}
W20260925 22:45:10.104496 20210 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:45:10.097735 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.198.2:56879 (local address 127.10.198.62:32879)
0925 22:45:10.097909 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0925 22:45:10.097913 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:45:10.098026 (+ 113us) server_negotiation.cc:407] Connection header received
0925 22:45:10.098088 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:45:10.098090 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:45:10.098149 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:45:10.098238 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:45:10.098711 (+ 473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:45:10.099582 (+ 871us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:45:10.100583 (+ 1001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:45:10.100835 (+ 252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:45:10.100934 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:45:10.104211 (+ 3277us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.198.2:56879: BlockingRecv error: recv got EOF from 127.10.198.2:56879 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
Sep 25 22:45:10 dist-test-slave-5vdr krb5kdc[19188](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.198.3@KRBTEST.COM for oryx/127.10.198.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 22:45:10 dist-test-slave-5vdr krb5kdc[19188](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.198.3@KRBTEST.COM for oryx/127.10.198.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 22:45:10.328938 20212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:45:10.322930 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.198.62:32879 (local address 127.10.198.3:50611)
0925 22:45:10.323243 (+ 313us) negotiation.cc:107] Waiting for socket to connect
0925 22:45:10.323258 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 22:45:10.323368 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:45:10.323632 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:45:10.323634 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:45:10.323819 (+ 185us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:45:10.324000 (+ 181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:45:10.324006 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:45:10.324838 (+ 832us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:45:10.324840 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:45:10.325552 (+ 712us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:45:10.325564 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:45:10.326171 (+ 607us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:45:10.326175 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:45:10.326263 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:45:10.326863 (+ 600us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:45:10.326875 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:45:10.328805 (+ 1930us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.198.62:32879: Server oryx/127.10.198.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":219,"thread_start_us":128,"threads_started":1}
W20260925 22:45:10.329238 20210 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:45:10.323105 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.198.3:50611 (local address 127.10.198.62:32879)
0925 22:45:10.323247 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0925 22:45:10.323250 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 22:45:10.323338 (+ 88us) server_negotiation.cc:407] Connection header received
0925 22:45:10.323457 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:45:10.323459 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:45:10.323506 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:45:10.323592 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:45:10.324117 (+ 525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:45:10.324717 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:45:10.325680 (+ 963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:45:10.326034 (+ 354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:45:10.326145 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:45:10.329002 (+ 2857us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.198.3:50611: BlockingRecv error: recv got EOF from 127.10.198.3:50611 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
Sep 25 22:45:10 dist-test-slave-5vdr krb5kdc[19188](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.198.1@KRBTEST.COM for oryx/127.10.198.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 22:45:10 dist-test-slave-5vdr krb5kdc[19188](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.198.1@KRBTEST.COM for oryx/127.10.198.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 22:45:10.907976 20214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:45:10.900842 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.198.62:32879 (local address 127.10.198.1:41457)
0925 22:45:10.901221 (+ 379us) negotiation.cc:107] Waiting for socket to connect
0925 22:45:10.901239 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0925 22:45:10.901348 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:45:10.901671 (+ 323us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:45:10.901673 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:45:10.901885 (+ 212us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:45:10.901989 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:45:10.901994 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:45:10.902886 (+ 892us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:45:10.902889 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:45:10.903567 (+ 678us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:45:10.903577 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:45:10.904371 (+ 794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:45:10.904374 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:45:10.904446 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:45:10.905202 (+ 756us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:45:10.905219 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:45:10.907749 (+ 2530us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.198.62:32879: Server oryx/127.10.198.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":300,"thread_start_us":107,"threads_started":1}
W20260925 22:45:10.908221 20213 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:45:10.900904 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.198.1:41457 (local address 127.10.198.62:32879)
0925 22:45:10.901190 (+ 286us) server_negotiation.cc:206] Beginning negotiation
0925 22:45:10.901194 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:45:10.901365 (+ 171us) server_negotiation.cc:407] Connection header received
0925 22:45:10.901430 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:45:10.901433 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:45:10.901479 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:45:10.901569 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:45:10.902149 (+ 580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:45:10.902731 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:45:10.903749 (+ 1018us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:45:10.904108 (+ 359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:45:10.904364 (+ 256us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:45:10.908033 (+ 3669us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.198.1:41457: BlockingRecv error: recv got EOF from 127.10.198.1:41457 (error 108)
Metrics: {"server-negotiator.queue_time_us":206,"thread_start_us":93,"threads_started":1}
Sep 25 22:45:11 dist-test-slave-5vdr krb5kdc[19188](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.198.2@KRBTEST.COM for oryx/127.10.198.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 22:45:11 dist-test-slave-5vdr krb5kdc[19188](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.198.2@KRBTEST.COM for oryx/127.10.198.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 22:45:11.112218 20215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:45:11.105261 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.198.62:32879 (local address 127.10.198.2:54517)
0925 22:45:11.105609 (+ 348us) negotiation.cc:107] Waiting for socket to connect
0925 22:45:11.105648 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0925 22:45:11.105779 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:45:11.106133 (+ 354us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:45:11.106136 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:45:11.106551 (+ 415us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:45:11.106688 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:45:11.106724 (+ 36us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:45:11.107505 (+ 781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:45:11.107508 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:45:11.108223 (+ 715us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:45:11.108236 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:45:11.108831 (+ 595us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:45:11.108834 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:45:11.108917 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:45:11.109500 (+ 583us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:45:11.109516 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:45:11.112058 (+ 2542us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.198.62:32879: Server oryx/127.10.198.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":255,"thread_start_us":87,"threads_started":1}
W20260925 22:45:11.112658 20213 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:45:11.105371 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.198.2:54517 (local address 127.10.198.62:32879)
0925 22:45:11.105740 (+ 369us) server_negotiation.cc:206] Beginning negotiation
0925 22:45:11.105744 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:45:11.105794 (+ 50us) server_negotiation.cc:407] Connection header received
0925 22:45:11.105897 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:45:11.105901 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:45:11.105967 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:45:11.106100 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:45:11.106817 (+ 717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:45:11.107359 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:45:11.108406 (+ 1047us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:45:11.108667 (+ 261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:45:11.108780 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:45:11.112355 (+ 3575us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.198.2:54517: BlockingRecv error: recv got EOF from 127.10.198.2:54517 (error 108)
Metrics: {"server-negotiator.queue_time_us":254}
Sep 25 22:45:11 dist-test-slave-5vdr krb5kdc[19188](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.198.3@KRBTEST.COM for oryx/127.10.198.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 22:45:11 dist-test-slave-5vdr krb5kdc[19188](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.198.3@KRBTEST.COM for oryx/127.10.198.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 22:45:11.335407 20216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:45:11.329774 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.198.62:32879 (local address 127.10.198.3:49573)
0925 22:45:11.330092 (+ 318us) negotiation.cc:107] Waiting for socket to connect
0925 22:45:11.330105 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 22:45:11.330202 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:45:11.330422 (+ 220us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:45:11.330424 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:45:11.330756 (+ 332us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:45:11.330853 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:45:11.330859 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:45:11.331619 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:45:11.331622 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:45:11.332224 (+ 602us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:45:11.332233 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:45:11.332694 (+ 461us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:45:11.332698 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:45:11.332776 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:45:11.333327 (+ 551us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:45:11.333339 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:45:11.335281 (+ 1942us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.198.62:32879: Server oryx/127.10.198.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":233,"thread_start_us":122,"threads_started":1}
W20260925 22:45:11.335685 20213 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:45:11.329903 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.198.3:49573 (local address 127.10.198.62:32879)
0925 22:45:11.330064 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0925 22:45:11.330067 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 22:45:11.330214 (+ 147us) server_negotiation.cc:407] Connection header received
0925 22:45:11.330263 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:45:11.330266 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:45:11.330314 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:45:11.330408 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:45:11.330973 (+ 565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:45:11.331500 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:45:11.332324 (+ 824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:45:11.332553 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:45:11.332635 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:45:11.335562 (+ 2927us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.198.3:49573: BlockingRecv error: recv got EOF from 127.10.198.3:49573 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
Sep 25 22:45:11 dist-test-slave-5vdr krb5kdc[19188](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.198.1@KRBTEST.COM for oryx/127.10.198.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 22:45:11 dist-test-slave-5vdr krb5kdc[19188](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.198.1@KRBTEST.COM for oryx/127.10.198.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 22:45:11.916213 20217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:45:11.909217 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.198.62:32879 (local address 127.10.198.1:35841)
0925 22:45:11.909516 (+ 299us) negotiation.cc:107] Waiting for socket to connect
0925 22:45:11.909546 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0925 22:45:11.909647 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:45:11.910189 (+ 542us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:45:11.910191 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:45:11.910639 (+ 448us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:45:11.910760 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:45:11.910767 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:45:11.911742 (+ 975us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:45:11.911745 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:45:11.912369 (+ 624us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:45:11.912378 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:45:11.912894 (+ 516us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:45:11.912897 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:45:11.912958 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:45:11.913579 (+ 621us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:45:11.913592 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:45:11.916026 (+ 2434us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.198.62:32879: Server oryx/127.10.198.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":207,"thread_start_us":114,"threads_started":1}
W20260925 22:45:11.916293 20218 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:45:11.909495 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.198.1:35841 (local address 127.10.198.62:32879)
0925 22:45:11.909739 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0925 22:45:11.909743 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:45:11.909760 (+ 17us) server_negotiation.cc:407] Connection header received
0925 22:45:11.909804 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:45:11.909806 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:45:11.909847 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:45:11.909941 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:45:11.911023 (+ 1082us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:45:11.911604 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:45:11.912505 (+ 901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:45:11.912756 (+ 251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:45:11.912873 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:45:11.916166 (+ 3293us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.198.1:35841: BlockingRecv error: recv got EOF from 127.10.198.1:35841 (error 108)
Metrics: {"server-negotiator.queue_time_us":171,"thread_start_us":94,"threads_started":1}
Sep 25 22:45:12 dist-test-slave-5vdr krb5kdc[19188](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.198.2@KRBTEST.COM for oryx/127.10.198.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 22:45:12 dist-test-slave-5vdr krb5kdc[19188](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.198.2@KRBTEST.COM for oryx/127.10.198.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 22:45:12.119622 20219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:45:12.113322 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.198.62:32879 (local address 127.10.198.2:60513)
0925 22:45:12.113700 (+ 378us) negotiation.cc:107] Waiting for socket to connect
0925 22:45:12.113714 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 22:45:12.113816 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:45:12.114082 (+ 266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:45:12.114085 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:45:12.114267 (+ 182us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:45:12.114362 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:45:12.114368 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:45:12.115279 (+ 911us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:45:12.115283 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:45:12.116177 (+ 894us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:45:12.116192 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:45:12.116769 (+ 577us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:45:12.116774 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:45:12.116889 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:45:12.117443 (+ 554us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:45:12.117455 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:45:12.119478 (+ 2023us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.198.62:32879: Server oryx/127.10.198.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":292,"thread_start_us":204,"threads_started":1}
W20260925 22:45:12.119786 20218 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:45:12.113503 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.198.2:60513 (local address 127.10.198.62:32879)
0925 22:45:12.113670 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0925 22:45:12.113674 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:45:12.113831 (+ 157us) server_negotiation.cc:407] Connection header received
0925 22:45:12.113884 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:45:12.113886 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:45:12.113933 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:45:12.114044 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:45:12.114483 (+ 439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:45:12.115159 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:45:12.116347 (+ 1188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:45:12.116584 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:45:12.116757 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:45:12.119652 (+ 2895us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.198.2:60513: BlockingRecv error: recv got EOF from 127.10.198.2:60513 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
Sep 25 22:45:12 dist-test-slave-5vdr krb5kdc[19188](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.198.3@KRBTEST.COM for oryx/127.10.198.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 22:45:12 dist-test-slave-5vdr krb5kdc[19188](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.198.3@KRBTEST.COM for oryx/127.10.198.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 22:45:12.342537 20220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:45:12.336318 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.198.62:32879 (local address 127.10.198.3:32849)
0925 22:45:12.336622 (+ 304us) negotiation.cc:107] Waiting for socket to connect
0925 22:45:12.336646 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0925 22:45:12.336784 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:45:12.337102 (+ 318us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:45:12.337106 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:45:12.337306 (+ 200us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:45:12.337510 (+ 204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:45:12.337516 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:45:12.338333 (+ 817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:45:12.338336 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:45:12.339041 (+ 705us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:45:12.339051 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:45:12.339506 (+ 455us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:45:12.339508 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:45:12.339627 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:45:12.340272 (+ 645us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:45:12.340291 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:45:12.342387 (+ 2096us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.198.62:32879: Server oryx/127.10.198.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":209,"thread_start_us":121,"threads_started":1}
W20260925 22:45:12.342715 20218 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:45:12.336530 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.198.3:32849 (local address 127.10.198.62:32879)
0925 22:45:12.336660 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0925 22:45:12.336664 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:45:12.336701 (+ 37us) server_negotiation.cc:407] Connection header received
0925 22:45:12.336903 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:45:12.336905 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:45:12.336952 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:45:12.337032 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:45:12.337671 (+ 639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:45:12.338218 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:45:12.339157 (+ 939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:45:12.339374 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:45:12.339460 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:45:12.342578 (+ 3118us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.198.3:32849: BlockingRecv error: recv got EOF from 127.10.198.3:32849 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
Sep 25 22:45:12 dist-test-slave-5vdr krb5kdc[19188](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.198.1@KRBTEST.COM for oryx/127.10.198.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 22:45:12 dist-test-slave-5vdr krb5kdc[19188](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.198.1@KRBTEST.COM for oryx/127.10.198.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 22:45:12.923255 20221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:45:12.917202 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.198.62:32879 (local address 127.10.198.1:58509)
0925 22:45:12.917667 (+ 465us) negotiation.cc:107] Waiting for socket to connect
0925 22:45:12.917682 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 22:45:12.917780 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:45:12.918054 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:45:12.918056 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:45:12.918314 (+ 258us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:45:12.918460 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:45:12.918467 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:45:12.919259 (+ 792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:45:12.919262 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:45:12.919881 (+ 619us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:45:12.919892 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:45:12.920381 (+ 489us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:45:12.920384 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:45:12.920454 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:45:12.921107 (+ 653us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:45:12.921120 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:45:12.923127 (+ 2007us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.198.62:32879: Server oryx/127.10.198.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":254,"thread_start_us":112,"threads_started":1}
W20260925 22:45:12.923554 20222 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:45:12.917430 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.198.1:58509 (local address 127.10.198.62:32879)
0925 22:45:12.917713 (+ 283us) server_negotiation.cc:206] Beginning negotiation
0925 22:45:12.917716 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 22:45:12.917732 (+ 16us) server_negotiation.cc:407] Connection header received
0925 22:45:12.917885 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:45:12.917888 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:45:12.917933 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:45:12.918015 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:45:12.918589 (+ 574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:45:12.919144 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:45:12.920007 (+ 863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:45:12.920234 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:45:12.920326 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:45:12.923412 (+ 3086us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.198.1:58509: BlockingRecv error: recv got EOF from 127.10.198.1:58509 (error 108)
Metrics: {"server-negotiator.queue_time_us":203,"thread_start_us":83,"threads_started":1}
Sep 25 22:45:13 dist-test-slave-5vdr krb5kdc[19188](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.198.2@KRBTEST.COM for oryx/127.10.198.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 22:45:13 dist-test-slave-5vdr krb5kdc[19188](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.198.2@KRBTEST.COM for oryx/127.10.198.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 22:45:13.126242 20223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:45:13.120531 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.198.62:32879 (local address 127.10.198.2:54647)
0925 22:45:13.120811 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0925 22:45:13.120826 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 22:45:13.120938 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:45:13.121177 (+ 239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:45:13.121180 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:45:13.121367 (+ 187us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:45:13.121463 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:45:13.121469 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:45:13.122270 (+ 801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:45:13.122273 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:45:13.122866 (+ 593us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:45:13.122875 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:45:13.123323 (+ 448us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:45:13.123326 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:45:13.123390 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:45:13.124040 (+ 650us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:45:13.124053 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:45:13.126092 (+ 2039us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.198.62:32879: Server oryx/127.10.198.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":81,"threads_started":1}
W20260925 22:45:13.126473 20222 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:45:13.120665 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.198.2:54647 (local address 127.10.198.62:32879)
0925 22:45:13.120833 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0925 22:45:13.120837 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:45:13.120909 (+ 72us) server_negotiation.cc:407] Connection header received
0925 22:45:13.121022 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:45:13.121024 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:45:13.121069 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:45:13.121137 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:45:13.121608 (+ 471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:45:13.122137 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:45:13.122989 (+ 852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:45:13.123210 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:45:13.123293 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:45:13.126324 (+ 3031us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.198.2:54647: BlockingRecv error: recv got EOF from 127.10.198.2:54647 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
Sep 25 22:45:13 dist-test-slave-5vdr krb5kdc[19188](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.198.3@KRBTEST.COM for oryx/127.10.198.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 22:45:13 dist-test-slave-5vdr krb5kdc[19188](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.198.3@KRBTEST.COM for oryx/127.10.198.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 22:45:13.349982 20224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:45:13.343506 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.198.62:32879 (local address 127.10.198.3:37175)
0925 22:45:13.343887 (+ 381us) negotiation.cc:107] Waiting for socket to connect
0925 22:45:13.343905 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0925 22:45:13.344060 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:45:13.344340 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:45:13.344343 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:45:13.344549 (+ 206us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:45:13.344649 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:45:13.344655 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:45:13.345425 (+ 770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:45:13.345430 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:45:13.346122 (+ 692us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:45:13.346134 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:45:13.346622 (+ 488us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:45:13.346626 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:45:13.346718 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:45:13.347261 (+ 543us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:45:13.347273 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:45:13.349819 (+ 2546us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.198.62:32879: Server oryx/127.10.198.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":267,"thread_start_us":129,"threads_started":1}
W20260925 22:45:13.350169 20222 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:45:13.343703 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.198.3:37175 (local address 127.10.198.62:32879)
0925 22:45:13.343889 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0925 22:45:13.343895 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 22:45:13.343997 (+ 102us) server_negotiation.cc:407] Connection header received
0925 22:45:13.344170 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:45:13.344175 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:45:13.344227 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:45:13.344316 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:45:13.344770 (+ 454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:45:13.345303 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:45:13.346255 (+ 952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:45:13.346495 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:45:13.346611 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:45:13.349997 (+ 3386us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.198.3:37175: BlockingRecv error: recv got EOF from 127.10.198.3:37175 (error 108)
Metrics: {"server-negotiator.queue_time_us":80}
Sep 25 22:45:13 dist-test-slave-5vdr krb5kdc[19188](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.198.1@KRBTEST.COM for oryx/127.10.198.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 22:45:13 dist-test-slave-5vdr krb5kdc[19188](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.198.1@KRBTEST.COM for oryx/127.10.198.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 22:45:13.930763 20225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:45:13.924349 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.198.62:32879 (local address 127.10.198.1:34911)
0925 22:45:13.924697 (+ 348us) negotiation.cc:107] Waiting for socket to connect
0925 22:45:13.924712 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 22:45:13.924812 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:45:13.925096 (+ 284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:45:13.925098 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:45:13.925312 (+ 214us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:45:13.925417 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:45:13.925423 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:45:13.926244 (+ 821us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:45:13.926247 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:45:13.926824 (+ 577us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:45:13.926832 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:45:13.927296 (+ 464us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:45:13.927299 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:45:13.927386 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:45:13.928085 (+ 699us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:45:13.928098 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:45:13.930540 (+ 2442us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.198.62:32879: Server oryx/127.10.198.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":241,"thread_start_us":77,"threads_started":1}
W20260925 22:45:13.930869 20226 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:45:13.924445 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.198.1:34911 (local address 127.10.198.62:32879)
0925 22:45:13.924755 (+ 310us) server_negotiation.cc:206] Beginning negotiation
0925 22:45:13.924758 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 22:45:13.924774 (+ 16us) server_negotiation.cc:407] Connection header received
0925 22:45:13.924917 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:45:13.924920 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:45:13.924966 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:45:13.925050 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:45:13.925546 (+ 496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:45:13.926124 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:45:13.926935 (+ 811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:45:13.927159 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:45:13.927239 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:45:13.930733 (+ 3494us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.198.1:34911: BlockingRecv error: recv got EOF from 127.10.198.1:34911 (error 108)
Metrics: {"server-negotiator.queue_time_us":237,"thread_start_us":130,"threads_started":1}
Sep 25 22:45:14 dist-test-slave-5vdr krb5kdc[19188](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.198.2@KRBTEST.COM for oryx/127.10.198.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 22:45:14 dist-test-slave-5vdr krb5kdc[19188](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.198.2@KRBTEST.COM for oryx/127.10.198.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 22:45:14.133662 20227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:45:14.127270 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.198.62:32879 (local address 127.10.198.2:33489)
0925 22:45:14.127641 (+ 371us) negotiation.cc:107] Waiting for socket to connect
0925 22:45:14.127657 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 22:45:14.127798 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:45:14.128137 (+ 339us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:45:14.128140 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:45:14.128367 (+ 227us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:45:14.128460 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:45:14.128466 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:45:14.129272 (+ 806us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:45:14.129278 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:45:14.129872 (+ 594us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:45:14.129881 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:45:14.130373 (+ 492us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:45:14.130377 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:45:14.130464 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:45:14.131036 (+ 572us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:45:14.131050 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:45:14.133470 (+ 2420us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.198.62:32879: Server oryx/127.10.198.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":284,"thread_start_us":98,"threads_started":1}
W20260925 22:45:14.133728 20226 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:45:14.127422 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.198.2:33489 (local address 127.10.198.62:32879)
0925 22:45:14.127613 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0925 22:45:14.127617 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:45:14.127753 (+ 136us) server_negotiation.cc:407] Connection header received
0925 22:45:14.127951 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:45:14.127956 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:45:14.128014 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:45:14.128110 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:45:14.128593 (+ 483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:45:14.129149 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:45:14.130008 (+ 859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:45:14.130238 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:45:14.130356 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:45:14.133618 (+ 3262us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.198.2:33489: BlockingRecv error: recv got EOF from 127.10.198.2:33489 (error 108)
Metrics: {"server-negotiator.queue_time_us":84}
Sep 25 22:45:14 dist-test-slave-5vdr krb5kdc[19188](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.198.3@KRBTEST.COM for oryx/127.10.198.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 22:45:14 dist-test-slave-5vdr krb5kdc[19188](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.198.3@KRBTEST.COM for oryx/127.10.198.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 22:45:14.356834 20228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:45:14.350895 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.198.62:32879 (local address 127.10.198.3:35691)
0925 22:45:14.351168 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0925 22:45:14.351182 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 22:45:14.351270 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:45:14.351548 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:45:14.351551 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:45:14.351729 (+ 178us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:45:14.351823 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:45:14.351829 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:45:14.352709 (+ 880us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:45:14.352712 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:45:14.353308 (+ 596us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:45:14.353319 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:45:14.353837 (+ 518us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:45:14.353840 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:45:14.353904 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:45:14.354476 (+ 572us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:45:14.354494 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:45:14.356710 (+ 2216us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.198.62:32879: Server oryx/127.10.198.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":102,"threads_started":1}
W20260925 22:45:14.356973 20226 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:45:14.351136 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.198.3:35691 (local address 127.10.198.62:32879)
0925 22:45:14.351277 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0925 22:45:14.351281 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:45:14.351327 (+ 46us) server_negotiation.cc:407] Connection header received
0925 22:45:14.351371 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:45:14.351383 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:45:14.351434 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:45:14.351504 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:45:14.352013 (+ 509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:45:14.352558 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:45:14.353458 (+ 900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:45:14.353679 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:45:14.353825 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:45:14.356869 (+ 3044us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.198.3:35691: BlockingRecv error: recv got EOF from 127.10.198.3:35691 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
Sep 25 22:45:14 dist-test-slave-5vdr krb5kdc[19188](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.198.1@KRBTEST.COM for oryx/127.10.198.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 22:45:14 dist-test-slave-5vdr krb5kdc[19188](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.198.1@KRBTEST.COM for oryx/127.10.198.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 22:45:14.943614 20229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:45:14.931703 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.198.62:32879 (local address 127.10.198.1:41047)
0925 22:45:14.932017 (+ 314us) negotiation.cc:107] Waiting for socket to connect
0925 22:45:14.932032 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 22:45:14.932137 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:45:14.932471 (+ 334us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:45:14.932473 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:45:14.932655 (+ 182us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:45:14.932797 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:45:14.932802 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:45:14.933579 (+ 777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:45:14.933581 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:45:14.934196 (+ 615us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:45:14.934205 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:45:14.934643 (+ 438us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:45:14.934645 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:45:14.934708 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:45:14.941055 (+ 6347us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:45:14.941073 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:45:14.943486 (+ 2413us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.198.62:32879: Server oryx/127.10.198.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":228,"thread_start_us":71,"threads_started":1}
W20260925 22:45:14.943825 20230 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:45:14.931760 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.198.1:41047 (local address 127.10.198.62:32879)
0925 22:45:14.932051 (+ 291us) server_negotiation.cc:206] Beginning negotiation
0925 22:45:14.932054 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 22:45:14.932088 (+ 34us) server_negotiation.cc:407] Connection header received
0925 22:45:14.932262 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:45:14.932265 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:45:14.932323 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:45:14.932411 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:45:14.932910 (+ 499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:45:14.933463 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:45:14.934317 (+ 854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:45:14.934539 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:45:14.934618 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:45:14.943656 (+ 9038us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.198.1:41047: BlockingRecv error: recv got EOF from 127.10.198.1:41047 (error 108)
Metrics: {"server-negotiator.queue_time_us":215,"thread_start_us":132,"threads_started":1}
Sep 25 22:45:15 dist-test-slave-5vdr krb5kdc[19188](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.198.2@KRBTEST.COM for oryx/127.10.198.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 22:45:15 dist-test-slave-5vdr krb5kdc[19188](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.198.2@KRBTEST.COM for oryx/127.10.198.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 22:45:15.141505 20231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:45:15.134764 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.198.62:32879 (local address 127.10.198.2:41305)
0925 22:45:15.135111 (+ 347us) negotiation.cc:107] Waiting for socket to connect
0925 22:45:15.135132 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0925 22:45:15.135258 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:45:15.135580 (+ 322us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:45:15.135583 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:45:15.135819 (+ 236us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:45:15.135989 (+ 170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:45:15.135995 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:45:15.136782 (+ 787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:45:15.136785 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:45:15.137621 (+ 836us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:45:15.137632 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:45:15.138339 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:45:15.138343 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:45:15.138428 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:45:15.139133 (+ 705us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:45:15.139149 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:45:15.141311 (+ 2162us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.198.62:32879: Server oryx/127.10.198.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":218,"thread_start_us":128,"threads_started":1}
W20260925 22:45:15.141613 20230 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:45:15.135039 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.198.2:41305 (local address 127.10.198.62:32879)
0925 22:45:15.135214 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0925 22:45:15.135218 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:45:15.135234 (+ 16us) server_negotiation.cc:407] Connection header received
0925 22:45:15.135383 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:45:15.135387 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:45:15.135442 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:45:15.135549 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:45:15.136111 (+ 562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:45:15.136652 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:45:15.137804 (+ 1152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:45:15.138206 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:45:15.138352 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:45:15.141482 (+ 3130us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.198.2:41305: BlockingRecv error: recv got EOF from 127.10.198.2:41305 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
Sep 25 22:45:15 dist-test-slave-5vdr krb5kdc[19188](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.198.3@KRBTEST.COM for oryx/127.10.198.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 22:45:15 dist-test-slave-5vdr krb5kdc[19188](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.198.3@KRBTEST.COM for oryx/127.10.198.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 22:45:15.363555 20232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:45:15.357818 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.198.62:32879 (local address 127.10.198.3:50401)
0925 22:45:15.358101 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0925 22:45:15.358115 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 22:45:15.358216 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:45:15.358485 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:45:15.358487 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:45:15.358665 (+ 178us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:45:15.358759 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:45:15.358764 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:45:15.359503 (+ 739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:45:15.359506 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:45:15.360166 (+ 660us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:45:15.360175 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:45:15.360600 (+ 425us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:45:15.360602 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:45:15.360662 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:45:15.361199 (+ 537us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:45:15.361215 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:45:15.363432 (+ 2217us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.198.62:32879: Server oryx/127.10.198.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":199,"thread_start_us":124,"threads_started":1}
W20260925 22:45:15.363782 20230 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:45:15.357929 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.198.3:50401 (local address 127.10.198.62:32879)
0925 22:45:15.358071 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0925 22:45:15.358075 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:45:15.358184 (+ 109us) server_negotiation.cc:407] Connection header received
0925 22:45:15.358296 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:45:15.358299 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:45:15.358346 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:45:15.358442 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:45:15.358863 (+ 421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:45:15.359392 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:45:15.360278 (+ 886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:45:15.360497 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:45:15.360579 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:45:15.363661 (+ 3082us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.198.3:50401: BlockingRecv error: recv got EOF from 127.10.198.3:50401 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Sep 25 22:45:15 dist-test-slave-5vdr krb5kdc[19188](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.198.1@KRBTEST.COM for oryx/127.10.198.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 22:45:15 dist-test-slave-5vdr krb5kdc[19188](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.198.1@KRBTEST.COM for oryx/127.10.198.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 22:45:15.951272 20233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:45:15.944721 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.198.62:32879 (local address 127.10.198.1:57981)
0925 22:45:15.945064 (+ 343us) negotiation.cc:107] Waiting for socket to connect
0925 22:45:15.945084 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0925 22:45:15.945208 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:45:15.945491 (+ 283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:45:15.945493 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:45:15.945680 (+ 187us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:45:15.945797 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:45:15.945804 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:45:15.946670 (+ 866us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:45:15.946672 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:45:15.947499 (+ 827us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:45:15.947511 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:45:15.948105 (+ 594us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:45:15.948109 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:45:15.948248 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:45:15.949022 (+ 774us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:45:15.949036 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:45:15.951141 (+ 2105us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.198.62:32879: Server oryx/127.10.198.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":236,"thread_start_us":78,"threads_started":1}
W20260925 22:45:15.951633 20234 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:45:15.944897 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.198.1:57981 (local address 127.10.198.62:32879)
0925 22:45:15.945179 (+ 282us) server_negotiation.cc:206] Beginning negotiation
0925 22:45:15.945182 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 22:45:15.945199 (+ 17us) server_negotiation.cc:407] Connection header received
0925 22:45:15.945299 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:45:15.945302 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:45:15.945355 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:45:15.945441 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:45:15.945927 (+ 486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:45:15.946535 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:45:15.947622 (+ 1087us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:45:15.947967 (+ 345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:45:15.948077 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:45:15.951315 (+ 3238us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.198.1:57981: BlockingRecv error: recv got EOF from 127.10.198.1:57981 (error 108)
Metrics: {"server-negotiator.queue_time_us":205,"thread_start_us":101,"threads_started":1}
Sep 25 22:45:16 dist-test-slave-5vdr krb5kdc[19188](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.198.2@KRBTEST.COM for oryx/127.10.198.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 22:45:16 dist-test-slave-5vdr krb5kdc[19188](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.198.2@KRBTEST.COM for oryx/127.10.198.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 22:45:16.148605 20235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:45:16.142543 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.198.62:32879 (local address 127.10.198.2:49055)
0925 22:45:16.142902 (+ 359us) negotiation.cc:107] Waiting for socket to connect
0925 22:45:16.142919 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 22:45:16.143053 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:45:16.143456 (+ 403us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:45:16.143462 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:45:16.143710 (+ 248us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:45:16.143830 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:45:16.143841 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:45:16.144736 (+ 895us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:45:16.144739 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:45:16.145324 (+ 585us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:45:16.145332 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:45:16.145804 (+ 472us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:45:16.145806 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:45:16.145870 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:45:16.146503 (+ 633us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:45:16.146516 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:45:16.148478 (+ 1962us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.198.62:32879: Server oryx/127.10.198.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":265,"thread_start_us":103,"threads_started":1}
W20260925 22:45:16.148840 20234 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:45:16.142678 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.198.2:49055 (local address 127.10.198.62:32879)
0925 22:45:16.142848 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0925 22:45:16.142853 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:45:16.143035 (+ 182us) server_negotiation.cc:407] Connection header received
0925 22:45:16.143186 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:45:16.143190 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:45:16.143297 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:45:16.143390 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:45:16.144008 (+ 618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:45:16.144620 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:45:16.145456 (+ 836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:45:16.145685 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:45:16.145786 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:45:16.148701 (+ 2915us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.198.2:49055: BlockingRecv error: recv got EOF from 127.10.198.2:49055 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
Sep 25 22:45:16 dist-test-slave-5vdr krb5kdc[19188](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.198.3@KRBTEST.COM for oryx/127.10.198.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 22:45:16 dist-test-slave-5vdr krb5kdc[19188](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.198.3@KRBTEST.COM for oryx/127.10.198.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 22:45:16.370430 20236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:45:16.364426 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.198.62:32879 (local address 127.10.198.3:42947)
0925 22:45:16.364713 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0925 22:45:16.364725 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 22:45:16.364862 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:45:16.365147 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:45:16.365150 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:45:16.365362 (+ 212us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:45:16.365454 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:45:16.365459 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:45:16.366232 (+ 773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:45:16.366238 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:45:16.366805 (+ 567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:45:16.366813 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:45:16.367375 (+ 562us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:45:16.367379 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:45:16.367459 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:45:16.368203 (+ 744us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:45:16.368216 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:45:16.370301 (+ 2085us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.198.62:32879: Server oryx/127.10.198.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":204,"thread_start_us":93,"threads_started":1}
W20260925 22:45:16.370671 20234 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:45:16.364481 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.198.3:42947 (local address 127.10.198.62:32879)
0925 22:45:16.364653 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0925 22:45:16.364658 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:45:16.364804 (+ 146us) server_negotiation.cc:407] Connection header received
0925 22:45:16.364973 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:45:16.364977 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:45:16.365028 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:45:16.365125 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:45:16.365578 (+ 453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:45:16.366117 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:45:16.367007 (+ 890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:45:16.367248 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:45:16.367368 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:45:16.370546 (+ 3178us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.198.3:42947: BlockingRecv error: recv got EOF from 127.10.198.3:42947 (error 108)
Metrics: {"server-negotiator.queue_time_us":80}
Sep 25 22:45:16 dist-test-slave-5vdr krb5kdc[19188](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.198.1@KRBTEST.COM for oryx/127.10.198.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 22:45:16 dist-test-slave-5vdr krb5kdc[19188](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.198.1@KRBTEST.COM for oryx/127.10.198.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 22:45:16.958632 20237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:45:16.952248 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.198.62:32879 (local address 127.10.198.1:32993)
0925 22:45:16.952517 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0925 22:45:16.952533 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 22:45:16.952630 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:45:16.952895 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:45:16.952898 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:45:16.953080 (+ 182us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:45:16.953196 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:45:16.953201 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:45:16.954044 (+ 843us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:45:16.954046 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:45:16.954623 (+ 577us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:45:16.954632 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:45:16.955269 (+ 637us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:45:16.955273 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:45:16.955345 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:45:16.956050 (+ 705us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:45:16.956064 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:45:16.958501 (+ 2437us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.198.62:32879: Server oryx/127.10.198.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":96,"threads_started":1}
W20260925 22:45:16.958901 20238 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:45:16.952307 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.198.1:32993 (local address 127.10.198.62:32879)
0925 22:45:16.952657 (+ 350us) server_negotiation.cc:206] Beginning negotiation
0925 22:45:16.952661 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:45:16.952679 (+ 18us) server_negotiation.cc:407] Connection header received
0925 22:45:16.952730 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:45:16.952732 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:45:16.952774 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:45:16.952861 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:45:16.953354 (+ 493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:45:16.953921 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:45:16.954763 (+ 842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:45:16.955074 (+ 311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:45:16.955173 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:45:16.958696 (+ 3523us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.198.1:32993: BlockingRecv error: recv got EOF from 127.10.198.1:32993 (error 108)
Metrics: {"server-negotiator.queue_time_us":285,"thread_start_us":201,"threads_started":1}
Sep 25 22:45:17 dist-test-slave-5vdr krb5kdc[19188](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.198.2@KRBTEST.COM for oryx/127.10.198.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 22:45:17 dist-test-slave-5vdr krb5kdc[19188](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.198.2@KRBTEST.COM for oryx/127.10.198.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 22:45:17.155346 20239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:45:17.149467 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.198.62:32879 (local address 127.10.198.2:44807)
0925 22:45:17.149742 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0925 22:45:17.149762 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0925 22:45:17.149889 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:45:17.150137 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:45:17.150139 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:45:17.150323 (+ 184us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:45:17.150424 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:45:17.150430 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:45:17.151207 (+ 777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:45:17.151209 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:45:17.151831 (+ 622us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:45:17.151880 (+ 49us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:45:17.152448 (+ 568us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:45:17.152452 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:45:17.152520 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:45:17.153134 (+ 614us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:45:17.153146 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:45:17.155199 (+ 2053us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.198.62:32879: Server oryx/127.10.198.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":97,"threads_started":1}
W20260925 22:45:17.155552 20238 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:45:17.149602 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.198.2:44807 (local address 127.10.198.62:32879)
0925 22:45:17.149743 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0925 22:45:17.149750 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 22:45:17.149860 (+ 110us) server_negotiation.cc:407] Connection header received
0925 22:45:17.149970 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:45:17.149972 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:45:17.150018 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:45:17.150087 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:45:17.150537 (+ 450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:45:17.151088 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:45:17.151983 (+ 895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:45:17.152319 (+ 336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:45:17.152413 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:45:17.155408 (+ 2995us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.198.2:44807: BlockingRecv error: recv got EOF from 127.10.198.2:44807 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
Sep 25 22:45:17 dist-test-slave-5vdr krb5kdc[19188](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.198.3@KRBTEST.COM for oryx/127.10.198.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 22:45:17 dist-test-slave-5vdr krb5kdc[19188](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.198.3@KRBTEST.COM for oryx/127.10.198.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 22:45:17.377919 20240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:45:17.371318 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.198.62:32879 (local address 127.10.198.3:46937)
0925 22:45:17.371647 (+ 329us) negotiation.cc:107] Waiting for socket to connect
0925 22:45:17.371665 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0925 22:45:17.371798 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:45:17.372160 (+ 362us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:45:17.372163 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:45:17.372407 (+ 244us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:45:17.372539 (+ 132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:45:17.372550 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:45:17.373468 (+ 918us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:45:17.373471 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:45:17.374107 (+ 636us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:45:17.374116 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:45:17.374645 (+ 529us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:45:17.374648 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:45:17.374715 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:45:17.375447 (+ 732us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:45:17.375462 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:45:17.377757 (+ 2295us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.198.62:32879: Server oryx/127.10.198.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":222,"thread_start_us":122,"threads_started":1}
W20260925 22:45:17.378291 20238 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:45:17.371465 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.198.3:46937 (local address 127.10.198.62:32879)
0925 22:45:17.371629 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0925 22:45:17.371634 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:45:17.371770 (+ 136us) server_negotiation.cc:407] Connection header received
0925 22:45:17.371961 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:45:17.371964 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:45:17.372025 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:45:17.372133 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:45:17.372675 (+ 542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:45:17.373301 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:45:17.374249 (+ 948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:45:17.374524 (+ 275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:45:17.374618 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:45:17.378127 (+ 3509us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.198.3:46937: BlockingRecv error: recv got EOF from 127.10.198.3:46937 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Sep 25 22:45:17 dist-test-slave-5vdr krb5kdc[19188](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.198.1@KRBTEST.COM for oryx/127.10.198.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 22:45:17 dist-test-slave-5vdr krb5kdc[19188](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.198.1@KRBTEST.COM for oryx/127.10.198.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 22:45:17.965639 20241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:45:17.959697 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.198.62:32879 (local address 127.10.198.1:55171)
0925 22:45:17.960012 (+ 315us) negotiation.cc:107] Waiting for socket to connect
0925 22:45:17.960026 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 22:45:17.960124 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:45:17.960541 (+ 417us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:45:17.960544 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:45:17.960745 (+ 201us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:45:17.960856 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:45:17.960862 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:45:17.961665 (+ 803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:45:17.961668 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:45:17.962315 (+ 647us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:45:17.962326 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:45:17.962802 (+ 476us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:45:17.962806 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:45:17.962892 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:45:17.963467 (+ 575us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:45:17.963480 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:45:17.965453 (+ 1973us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.198.62:32879: Server oryx/127.10.198.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":222,"thread_start_us":107,"threads_started":1}
W20260925 22:45:17.965734 20242 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:45:17.959894 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.198.1:55171 (local address 127.10.198.62:32879)
0925 22:45:17.960220 (+ 326us) server_negotiation.cc:206] Beginning negotiation
0925 22:45:17.960224 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:45:17.960261 (+ 37us) server_negotiation.cc:407] Connection header received
0925 22:45:17.960319 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:45:17.960326 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:45:17.960372 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:45:17.960467 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:45:17.960976 (+ 509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:45:17.961524 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:45:17.962441 (+ 917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:45:17.962681 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:45:17.962797 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:45:17.965605 (+ 2808us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.198.1:55171: BlockingRecv error: recv got EOF from 127.10.198.1:55171 (error 108)
Metrics: {"server-negotiator.queue_time_us":229,"thread_start_us":84,"threads_started":1}
Sep 25 22:45:18 dist-test-slave-5vdr krb5kdc[19188](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.198.2@KRBTEST.COM for oryx/127.10.198.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 22:45:18 dist-test-slave-5vdr krb5kdc[19188](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.198.2@KRBTEST.COM for oryx/127.10.198.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 22:45:18.162365 20243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:45:18.156454 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.198.62:32879 (local address 127.10.198.2:51355)
0925 22:45:18.156771 (+ 317us) negotiation.cc:107] Waiting for socket to connect
0925 22:45:18.156795 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0925 22:45:18.156929 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:45:18.157216 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:45:18.157219 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:45:18.157441 (+ 222us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:45:18.157560 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:45:18.157569 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:45:18.158438 (+ 869us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:45:18.158441 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:45:18.159039 (+ 598us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:45:18.159048 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:45:18.159527 (+ 479us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:45:18.159529 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:45:18.159593 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:45:18.160191 (+ 598us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:45:18.160203 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:45:18.162211 (+ 2008us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.198.62:32879: Server oryx/127.10.198.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":240,"thread_start_us":108,"threads_started":1}
W20260925 22:45:18.162652 20242 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:45:18.156530 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.198.2:51355 (local address 127.10.198.62:32879)
0925 22:45:18.156694 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0925 22:45:18.156699 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:45:18.156881 (+ 182us) server_negotiation.cc:407] Connection header received
0925 22:45:18.157042 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:45:18.157047 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:45:18.157101 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:45:18.157192 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:45:18.157685 (+ 493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:45:18.158329 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:45:18.159164 (+ 835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:45:18.159417 (+ 253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:45:18.159502 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:45:18.162527 (+ 3025us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.198.2:51355: BlockingRecv error: recv got EOF from 127.10.198.2:51355 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
Sep 25 22:45:18 dist-test-slave-5vdr krb5kdc[19188](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.198.3@KRBTEST.COM for oryx/127.10.198.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 22:45:18 dist-test-slave-5vdr krb5kdc[19188](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.198.3@KRBTEST.COM for oryx/127.10.198.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 22:45:18.384814 20244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:45:18.378807 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.198.62:32879 (local address 127.10.198.3:49885)
0925 22:45:18.379164 (+ 357us) negotiation.cc:107] Waiting for socket to connect
0925 22:45:18.379178 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 22:45:18.379318 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:45:18.379609 (+ 291us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:45:18.379612 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:45:18.379822 (+ 210us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:45:18.379951 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:45:18.379956 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:45:18.380765 (+ 809us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:45:18.380772 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:45:18.381360 (+ 588us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:45:18.381368 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:45:18.381845 (+ 477us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:45:18.381849 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:45:18.381937 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:45:18.382507 (+ 570us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:45:18.382519 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:45:18.384644 (+ 2125us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.198.62:32879: Server oryx/127.10.198.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":260,"thread_start_us":104,"threads_started":1}
W20260925 22:45:18.384971 20242 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:45:18.378930 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.198.3:49885 (local address 127.10.198.62:32879)
0925 22:45:18.379107 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0925 22:45:18.379110 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 22:45:18.379269 (+ 159us) server_negotiation.cc:407] Connection header received
0925 22:45:18.379429 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:45:18.379433 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:45:18.379496 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:45:18.379586 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:45:18.380071 (+ 485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:45:18.380648 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:45:18.381486 (+ 838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:45:18.381717 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:45:18.381833 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:45:18.384844 (+ 3011us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.198.3:49885: BlockingRecv error: recv got EOF from 127.10.198.3:49885 (error 108)
Metrics: {"server-negotiator.queue_time_us":86}
Sep 25 22:45:18 dist-test-slave-5vdr krb5kdc[19188](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.198.1@KRBTEST.COM for oryx/127.10.198.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 22:45:18 dist-test-slave-5vdr krb5kdc[19188](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.198.1@KRBTEST.COM for oryx/127.10.198.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 22:45:18.972710 20245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:45:18.966820 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.198.62:32879 (local address 127.10.198.1:46459)
0925 22:45:18.967169 (+ 349us) negotiation.cc:107] Waiting for socket to connect
0925 22:45:18.967188 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0925 22:45:18.967299 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:45:18.967559 (+ 260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:45:18.967561 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:45:18.967755 (+ 194us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:45:18.967888 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:45:18.967894 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:45:18.968708 (+ 814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:45:18.968711 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:45:18.969277 (+ 566us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:45:18.969285 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:45:18.969737 (+ 452us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:45:18.969740 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:45:18.969800 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:45:18.970409 (+ 609us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:45:18.970422 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:45:18.972491 (+ 2069us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.198.62:32879: Server oryx/127.10.198.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":228,"thread_start_us":88,"threads_started":1}
W20260925 22:45:18.972889 20246 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:45:18.967020 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.198.1:46459 (local address 127.10.198.62:32879)
0925 22:45:18.967272 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0925 22:45:18.967275 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 22:45:18.967291 (+ 16us) server_negotiation.cc:407] Connection header received
0925 22:45:18.967390 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:45:18.967392 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:45:18.967436 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:45:18.967519 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:45:18.968014 (+ 495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:45:18.968584 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:45:18.969396 (+ 812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:45:18.969622 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:45:18.969715 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:45:18.972737 (+ 3022us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.198.1:46459: BlockingRecv error: recv got EOF from 127.10.198.1:46459 (error 108)
Metrics: {"server-negotiator.queue_time_us":169,"thread_start_us":78,"threads_started":1}
Sep 25 22:45:19 dist-test-slave-5vdr krb5kdc[19188](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.198.2@KRBTEST.COM for oryx/127.10.198.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 22:45:19 dist-test-slave-5vdr krb5kdc[19188](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.198.2@KRBTEST.COM for oryx/127.10.198.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 22:45:19.170320 20247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:45:19.163537 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.198.62:32879 (local address 127.10.198.2:35093)
0925 22:45:19.163898 (+ 361us) negotiation.cc:107] Waiting for socket to connect
0925 22:45:19.163917 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0925 22:45:19.164042 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:45:19.164338 (+ 296us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:45:19.164341 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:45:19.164551 (+ 210us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:45:19.164654 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:45:19.164659 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:45:19.165707 (+ 1048us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:45:19.165710 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:45:19.166414 (+ 704us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:45:19.166427 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:45:19.167045 (+ 618us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:45:19.167049 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:45:19.167121 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:45:19.167755 (+ 634us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:45:19.167771 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:45:19.170118 (+ 2347us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.198.62:32879: Server oryx/127.10.198.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":201,"thread_start_us":99,"threads_started":1}
W20260925 22:45:19.170373 20246 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:45:19.163651 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.198.2:35093 (local address 127.10.198.62:32879)
0925 22:45:19.163885 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0925 22:45:19.163889 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:45:19.164044 (+ 155us) server_negotiation.cc:407] Connection header received
0925 22:45:19.164139 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:45:19.164142 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:45:19.164191 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:45:19.164267 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:45:19.164963 (+ 696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:45:19.165530 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:45:19.166591 (+ 1061us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:45:19.166881 (+ 290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:45:19.166991 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:45:19.170257 (+ 3266us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.198.2:35093: BlockingRecv error: recv got EOF from 127.10.198.2:35093 (error 108)
Metrics: {"server-negotiator.queue_time_us":114}
I20260925 22:45:19.266964 11032 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrWXm8X/build/debug/bin/kudu with pid 19771
I20260925 22:45:19.272742 11032 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrWXm8X/build/debug/bin/kudu with pid 19923
I20260925 22:45:19.278437 11032 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrWXm8X/build/debug/bin/kudu with pid 20055
I20260925 22:45:19.284893 11032 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrWXm8X/build/debug/bin/kudu with pid 19701
2026-09-25T22:45:19Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (18237 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-09-25T22:45:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T22:45:19Z Disabled control of system clock
I20260925 22:45:19.317756 11032 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrWXm8X/build/debug/bin/kudu
/tmp/dist-test-taskrWXm8X/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790376259650642-11032-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790376259650642-11032-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790376259650642-11032-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790376259650642-11032-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.198.62:36299
--webserver_interface=127.10.198.62
--webserver_port=0
--builtin_ntp_servers=127.10.198.20:45809
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.198.62:36299
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260925 22:45:19.438325 20251 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:45:19.438673 20251 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:45:19.438711 20251 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:45:19.443032 20251 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 22:45:19.443123 20251 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:45:19.443140 20251 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:45:19.443156 20251 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 22:45:19.443172 20251 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 22:45:19.448644 20251 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.198.20:45809
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790376259650642-11032-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790376259650642-11032-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.198.62:36299
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.198.62:36299
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790376259650642-11032-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.198.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20251
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790376259650642-11032-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:09:16 UTC on ccycloud.quasar-afmhyw.root.comops.site
build id 4673379
I20260925 22:45:19.449823 20251 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:45:19.451256 20251 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:45:19.460826 20257 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:45:19.460702 20256 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:45:19.460701 20259 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:45:19.460753 20251 server_base.cc:1034] running on GCE node
I20260925 22:45:19.461755 20251 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:45:19.463102 20251 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:45:19.464321 20251 hybrid_clock.cc:648] HybridClock initialized: now 1790376319464296 us; error 45 us; skew 500 ppm
I20260925 22:45:19.466507 20251 webserver.cc:460] Webserver started at http://127.10.198.62:43989/ using document root <none> and password file <none>
I20260925 22:45:19.467196 20251 fs_manager.cc:359] Metadata directory not provided
I20260925 22:45:19.467314 20251 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:45:19.467587 20251 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:45:19.469964 20251 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790376259650642-11032-0/minicluster-data/master-0/data/instance:
uuid: "9217cb3a5d334f519b69316455002573"
format_stamp: "Formatted at 2026-09-25 22:45:19 on dist-test-slave-5vdr"
server_key: "05222e6d80caf59d17d0ee9872b9136d"
server_key_iv: "b9cb7c73a33303d62ab0ab66c0b23e5e"
server_key_version: "encryptionkey@0"
I20260925 22:45:19.470636 20251 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790376259650642-11032-0/minicluster-data/master-0/wal/instance:
uuid: "9217cb3a5d334f519b69316455002573"
format_stamp: "Formatted at 2026-09-25 22:45:19 on dist-test-slave-5vdr"
server_key: "05222e6d80caf59d17d0ee9872b9136d"
server_key_iv: "b9cb7c73a33303d62ab0ab66c0b23e5e"
server_key_version: "encryptionkey@0"
I20260925 22:45:19.474953 20251 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260925 22:45:19.477969 20265 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:45:19.479595 20251 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260925 22:45:19.479800 20251 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790376259650642-11032-0/minicluster-data/master-0/data,/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790376259650642-11032-0/minicluster-data/master-0/wal
uuid: "9217cb3a5d334f519b69316455002573"
format_stamp: "Formatted at 2026-09-25 22:45:19 on dist-test-slave-5vdr"
server_key: "05222e6d80caf59d17d0ee9872b9136d"
server_key_iv: "b9cb7c73a33303d62ab0ab66c0b23e5e"
server_key_version: "encryptionkey@0"
I20260925 22:45:19.480038 20251 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790376259650642-11032-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790376259650642-11032-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790376259650642-11032-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:45:19.494539 20251 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:45:19.495402 20251 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:45:19.495633 20251 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:45:19.503901 20251 rpc_server.cc:307] RPC server started. Bound to: 127.10.198.62:36299
I20260925 22:45:19.503886 20317 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.198.62:36299 every 8 connection(s)
I20260925 22:45:19.505154 20251 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790376259650642-11032-0/minicluster-data/master-0/data/info.pb
I20260925 22:45:19.509155 20318 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:45:19.516498 11032 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrWXm8X/build/debug/bin/kudu as pid 20251
I20260925 22:45:19.516911 11032 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790376259650642-11032-0/minicluster-data/master-0/wal/instance
I20260925 22:45:19.517920 11032 external_mini_cluster.cc:1428] Setting key 2f080447aae0dfb73dfac4b258933947
I20260925 22:45:19.518857 20318 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9217cb3a5d334f519b69316455002573: Bootstrap starting.
I20260925 22:45:19.522990 20318 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9217cb3a5d334f519b69316455002573: Neither blocks nor log segments found. Creating new log.
I20260925 22:45:19.524215 20318 log.cc:824] T 00000000000000000000000000000000 P 9217cb3a5d334f519b69316455002573: Log is configured to *not* fsync() on all Append() calls
I20260925 22:45:19.528587 20318 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9217cb3a5d334f519b69316455002573: No bootstrap required, opened a new log
I20260925 22:45:19.534214 20318 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9217cb3a5d334f519b69316455002573 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9217cb3a5d334f519b69316455002573" member_type: VOTER last_known_addr { host: "127.10.198.62" port: 36299 } }
I20260925 22:45:19.534545 20318 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9217cb3a5d334f519b69316455002573 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:45:19.534643 20318 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9217cb3a5d334f519b69316455002573 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9217cb3a5d334f519b69316455002573, State: Initialized, Role: FOLLOWER
I20260925 22:45:19.535225 20318 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9217cb3a5d334f519b69316455002573 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9217cb3a5d334f519b69316455002573" member_type: VOTER last_known_addr { host: "127.10.198.62" port: 36299 } }
I20260925 22:45:19.535512 20318 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9217cb3a5d334f519b69316455002573 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 22:45:19.535616 20318 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9217cb3a5d334f519b69316455002573 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 22:45:19.535738 20318 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9217cb3a5d334f519b69316455002573 [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:45:19.536808 20322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:45:19.525382 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37572 (local address 127.10.198.62:36299)
0925 22:45:19.526287 (+ 905us) server_negotiation.cc:206] Beginning negotiation
0925 22:45:19.526310 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0925 22:45:19.526361 (+ 51us) server_negotiation.cc:407] Connection header received
0925 22:45:19.527458 (+ 1097us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:45:19.527485 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:45:19.528035 (+ 550us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:45:19.529015 (+ 980us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:45:19.529407 (+ 392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:45:19.530588 (+ 1181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:45:19.532143 (+ 1555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:45:19.533003 (+ 860us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:45:19.533715 (+ 712us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:45:19.534280 (+ 565us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:45:19.534286 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:45:19.534306 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 22:45:19.534844 (+ 538us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:45:19.534920 (+ 76us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 22:45:19.534948 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:45:19.535441 (+ 493us) server_negotiation.cc:1035] Waiting for connection context
0925 22:45:19.535525 (+ 84us) server_negotiation.cc:299] Negotiation successful
0925 22:45:19.535871 (+ 346us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":410,"thread_start_us":141,"threads_started":1}
I20260925 22:45:19.536998 20318 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9217cb3a5d334f519b69316455002573 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9217cb3a5d334f519b69316455002573" member_type: VOTER last_known_addr { host: "127.10.198.62" port: 36299 } }
I20260925 22:45:19.537461 20318 leader_election.cc:302] T 00000000000000000000000000000000 P 9217cb3a5d334f519b69316455002573 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9217cb3a5d334f519b69316455002573; no voters:
I20260925 22:45:19.537935 20318 leader_election.cc:288] T 00000000000000000000000000000000 P 9217cb3a5d334f519b69316455002573 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 22:45:19.538213 20323 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9217cb3a5d334f519b69316455002573 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 22:45:19.538455 20323 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9217cb3a5d334f519b69316455002573 [term 1 LEADER]: Becoming Leader. State: Replica: 9217cb3a5d334f519b69316455002573, State: Running, Role: LEADER
I20260925 22:45:19.538803 20323 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9217cb3a5d334f519b69316455002573 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9217cb3a5d334f519b69316455002573" member_type: VOTER last_known_addr { host: "127.10.198.62" port: 36299 } }
I20260925 22:45:19.539614 20318 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9217cb3a5d334f519b69316455002573 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 22:45:19.540268 20323 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9217cb3a5d334f519b69316455002573 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9217cb3a5d334f519b69316455002573" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9217cb3a5d334f519b69316455002573" member_type: VOTER last_known_addr { host: "127.10.198.62" port: 36299 } } }
I20260925 22:45:19.540478 20323 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9217cb3a5d334f519b69316455002573 [sys.catalog]: This master's current role is: LEADER
I20260925 22:45:19.540757 20323 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9217cb3a5d334f519b69316455002573 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9217cb3a5d334f519b69316455002573. Latest consensus state: current_term: 1 leader_uuid: "9217cb3a5d334f519b69316455002573" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9217cb3a5d334f519b69316455002573" member_type: VOTER last_known_addr { host: "127.10.198.62" port: 36299 } } }
I20260925 22:45:19.540943 20323 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9217cb3a5d334f519b69316455002573 [sys.catalog]: This master's current role is: LEADER
I20260925 22:45:19.540998 20328 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 22:45:19.544991 20328 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 22:45:19.553346 20328 catalog_manager.cc:1347] Generated new cluster ID: ee07ff19cbdb4e0194c0cfe254cfb0e8
I20260925 22:45:19.553455 20328 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 22:45:19.565353 20328 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 22:45:19.566383 20328 catalog_manager.cc:1504] Loading token signing keys...
I20260925 22:45:19.575668 20328 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9217cb3a5d334f519b69316455002573: Generated new TSK 0
I20260925 22:45:19.576567 20328 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 22:45:19.584012 11032 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrWXm8X/build/debug/bin/kudu
/tmp/dist-test-taskrWXm8X/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790376259650642-11032-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790376259650642-11032-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790376259650642-11032-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790376259650642-11032-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.198.1:0
--local_ip_for_outbound_sockets=127.10.198.1
--webserver_interface=127.10.198.1
--webserver_port=0
--tserver_master_addrs=127.10.198.62:36299
--builtin_ntp_servers=127.10.198.20:45809
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260925 22:45:19.731962 20342 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:45:19.732306 20342 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:45:19.732378 20342 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:45:19.736835 20342 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:45:19.736923 20342 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:45:19.736982 20342 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.198.1
I20260925 22:45:19.742565 20342 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.198.20:45809
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790376259650642-11032-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790376259650642-11032-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.198.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790376259650642-11032-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.198.1
--webserver_port=0
--tserver_master_addrs=127.10.198.62:36299
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20342
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.198.1
--log_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790376259650642-11032-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:09:16 UTC on ccycloud.quasar-afmhyw.root.comops.site
build id 4673379
I20260925 22:45:19.743782 20342 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:45:19.745409 20342 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:45:19.756011 20347 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:45:19.756045 20350 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:45:19.756228 20342 server_base.cc:1034] running on GCE node
W20260925 22:45:19.756407 20348 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:45:19.756881 20342 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:45:19.757648 20342 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:45:19.758919 20342 hybrid_clock.cc:648] HybridClock initialized: now 1790376319758884 us; error 51 us; skew 500 ppm
I20260925 22:45:19.761449 20342 webserver.cc:460] Webserver started at http://127.10.198.1:39195/ using document root <none> and password file <none>
I20260925 22:45:19.762202 20342 fs_manager.cc:359] Metadata directory not provided
I20260925 22:45:19.762321 20342 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:45:19.762720 20342 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:45:19.765269 20342 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790376259650642-11032-0/minicluster-data/ts-0/data/instance:
uuid: "55d7b400f1344a27a9c6746c6a5c218a"
format_stamp: "Formatted at 2026-09-25 22:45:19 on dist-test-slave-5vdr"
server_key: "a209a7872ffb0bc183bede6ea95ad4ee"
server_key_iv: "6c44e74b3c2254552f441c12821db55d"
server_key_version: "encryptionkey@0"
I20260925 22:45:19.766182 20342 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790376259650642-11032-0/minicluster-data/ts-0/wal/instance:
uuid: "55d7b400f1344a27a9c6746c6a5c218a"
format_stamp: "Formatted at 2026-09-25 22:45:19 on dist-test-slave-5vdr"
server_key: "a209a7872ffb0bc183bede6ea95ad4ee"
server_key_iv: "6c44e74b3c2254552f441c12821db55d"
server_key_version: "encryptionkey@0"
I20260925 22:45:19.770720 20342 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260925 22:45:19.773908 20356 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:45:19.775431 20342 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260925 22:45:19.775653 20342 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790376259650642-11032-0/minicluster-data/ts-0/data,/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790376259650642-11032-0/minicluster-data/ts-0/wal
uuid: "55d7b400f1344a27a9c6746c6a5c218a"
format_stamp: "Formatted at 2026-09-25 22:45:19 on dist-test-slave-5vdr"
server_key: "a209a7872ffb0bc183bede6ea95ad4ee"
server_key_iv: "6c44e74b3c2254552f441c12821db55d"
server_key_version: "encryptionkey@0"
I20260925 22:45:19.775821 20342 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790376259650642-11032-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790376259650642-11032-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790376259650642-11032-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:45:19.790918 20342 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:45:19.791790 20342 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:45:19.792080 20342 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:45:19.792743 20342 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 22:45:19.793807 20342 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:45:19.793893 20342 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:45:19.793972 20342 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:45:19.794018 20342 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 22:45:19.803534 20342 rpc_server.cc:307] RPC server started. Bound to: 127.10.198.1:40807
I20260925 22:45:19.803579 20469 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.198.1:40807 every 8 connection(s)
I20260925 22:45:19.804705 20342 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790376259650642-11032-0/minicluster-data/ts-0/data/info.pb
I20260925 22:45:19.813041 11032 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrWXm8X/build/debug/bin/kudu as pid 20342
I20260925 22:45:19.813177 11032 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790376259650642-11032-0/minicluster-data/ts-0/wal/instance
I20260925 22:45:19.813485 11032 external_mini_cluster.cc:1428] Setting key 88238dad05d121eba994f4448370fec4
I20260925 22:45:19.816004 11032 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrWXm8X/build/debug/bin/kudu
/tmp/dist-test-taskrWXm8X/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790376259650642-11032-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790376259650642-11032-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790376259650642-11032-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790376259650642-11032-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.198.2:0
--local_ip_for_outbound_sockets=127.10.198.2
--webserver_interface=127.10.198.2
--webserver_port=0
--tserver_master_addrs=127.10.198.62:36299
--builtin_ntp_servers=127.10.198.20:45809
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260925 22:45:19.816787 20322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:45:19.806882 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.198.1:51599 (local address 127.10.198.62:36299)
0925 22:45:19.807039 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0925 22:45:19.807044 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:45:19.808610 (+ 1566us) server_negotiation.cc:407] Connection header received
0925 22:45:19.810078 (+ 1468us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:45:19.810088 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:45:19.810152 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:45:19.810245 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:45:19.811441 (+ 1196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:45:19.812181 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:45:19.814214 (+ 2033us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:45:19.814516 (+ 302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:45:19.814673 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:45:19.816082 (+ 1409us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:45:19.816087 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:45:19.816092 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 22:45:19.816110 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:45:19.816160 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 22:45:19.816173 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:45:19.816247 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0925 22:45:19.816469 (+ 222us) server_negotiation.cc:299] Negotiation successful
0925 22:45:19.816558 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260925 22:45:19.817647 20472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:45:19.807440 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.198.62:36299 (local address 127.10.198.1:51599)
0925 22:45:19.808415 (+ 975us) negotiation.cc:107] Waiting for socket to connect
0925 22:45:19.808468 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0925 22:45:19.809801 (+ 1333us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:45:19.810501 (+ 700us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:45:19.810517 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:45:19.810628 (+ 111us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:45:19.811230 (+ 602us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:45:19.811257 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:45:19.812362 (+ 1105us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:45:19.812366 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:45:19.813998 (+ 1632us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:45:19.814017 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:45:19.814728 (+ 711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:45:19.814734 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:45:19.814939 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:45:19.815707 (+ 768us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 22:45:19.815708 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:45:19.815758 (+ 50us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 22:45:19.815838 (+ 80us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:45:19.816293 (+ 455us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:45:19.816301 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:45:19.816306 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 22:45:19.816417 (+ 111us) client_negotiation.cc:240] Negotiation successful
0925 22:45:19.816698 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":450,"thread_start_us":242,"threads_started":1}
I20260925 22:45:19.819260 20470 heartbeater.cc:347] Connected to a master server at 127.10.198.62:36299
I20260925 22:45:19.819550 20470 heartbeater.cc:464] Registering TS with master...
I20260925 22:45:19.820240 20470 heartbeater.cc:511] Master 127.10.198.62:36299 requested a full tablet report, sending...
I20260925 22:45:19.822060 20282 ts_manager.cc:194] Registered new tserver with Master: 55d7b400f1344a27a9c6746c6a5c218a (127.10.198.1:40807)
I20260925 22:45:19.823338 20282 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.10.198.1:51599
W20260925 22:45:19.937875 20473 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:45:19.938233 20473 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:45:19.938314 20473 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:45:19.943054 20473 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:45:19.943154 20473 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:45:19.943250 20473 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.198.2
I20260925 22:45:19.948748 20473 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.198.20:45809
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790376259650642-11032-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790376259650642-11032-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.198.2:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790376259650642-11032-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.198.2
--webserver_port=0
--tserver_master_addrs=127.10.198.62:36299
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20473
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.198.2
--log_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790376259650642-11032-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:09:16 UTC on ccycloud.quasar-afmhyw.root.comops.site
build id 4673379
I20260925 22:45:19.949995 20473 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:45:19.951478 20473 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 22:45:19.963001 20473 server_base.cc:1034] running on GCE node
W20260925 22:45:19.963001 20478 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:45:19.963028 20481 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:45:19.963002 20479 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:45:19.963934 20473 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:45:19.964723 20473 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:45:19.965924 20473 hybrid_clock.cc:648] HybridClock initialized: now 1790376319965903 us; error 41 us; skew 500 ppm
I20260925 22:45:19.971702 20473 webserver.cc:460] Webserver started at http://127.10.198.2:44283/ using document root <none> and password file <none>
I20260925 22:45:19.972451 20473 fs_manager.cc:359] Metadata directory not provided
I20260925 22:45:19.972533 20473 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:45:19.972793 20473 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:45:19.975092 20473 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790376259650642-11032-0/minicluster-data/ts-1/data/instance:
uuid: "879e6ee7b81e417bbae0339eab6b21a1"
format_stamp: "Formatted at 2026-09-25 22:45:19 on dist-test-slave-5vdr"
server_key: "165c784175e91f5961c60fc5dba57d33"
server_key_iv: "fbff7b569baf07a062736b8b931807a3"
server_key_version: "encryptionkey@0"
I20260925 22:45:19.975751 20473 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790376259650642-11032-0/minicluster-data/ts-1/wal/instance:
uuid: "879e6ee7b81e417bbae0339eab6b21a1"
format_stamp: "Formatted at 2026-09-25 22:45:19 on dist-test-slave-5vdr"
server_key: "165c784175e91f5961c60fc5dba57d33"
server_key_iv: "fbff7b569baf07a062736b8b931807a3"
server_key_version: "encryptionkey@0"
I20260925 22:45:19.980034 20473 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260925 22:45:19.983063 20487 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:45:19.984642 20473 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 22:45:19.984817 20473 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790376259650642-11032-0/minicluster-data/ts-1/data,/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790376259650642-11032-0/minicluster-data/ts-1/wal
uuid: "879e6ee7b81e417bbae0339eab6b21a1"
format_stamp: "Formatted at 2026-09-25 22:45:19 on dist-test-slave-5vdr"
server_key: "165c784175e91f5961c60fc5dba57d33"
server_key_iv: "fbff7b569baf07a062736b8b931807a3"
server_key_version: "encryptionkey@0"
I20260925 22:45:19.984959 20473 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790376259650642-11032-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790376259650642-11032-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790376259650642-11032-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:45:20.014206 20473 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:45:20.014916 20473 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:45:20.015077 20473 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:45:20.015662 20473 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 22:45:20.016747 20473 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:45:20.016798 20473 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:45:20.016841 20473 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:45:20.016856 20473 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 22:45:20.026746 20473 rpc_server.cc:307] RPC server started. Bound to: 127.10.198.2:36195
I20260925 22:45:20.026770 20600 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.198.2:36195 every 8 connection(s)
I20260925 22:45:20.027823 20473 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790376259650642-11032-0/minicluster-data/ts-1/data/info.pb
I20260925 22:45:20.034461 11032 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrWXm8X/build/debug/bin/kudu as pid 20473
I20260925 22:45:20.034598 11032 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790376259650642-11032-0/minicluster-data/ts-1/wal/instance
I20260925 22:45:20.034899 11032 external_mini_cluster.cc:1428] Setting key 3c76526b5fc335734bec25eff18f5719
I20260925 22:45:20.037107 11032 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrWXm8X/build/debug/bin/kudu
/tmp/dist-test-taskrWXm8X/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790376259650642-11032-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790376259650642-11032-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790376259650642-11032-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790376259650642-11032-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.198.3:0
--local_ip_for_outbound_sockets=127.10.198.3
--webserver_interface=127.10.198.3
--webserver_port=0
--tserver_master_addrs=127.10.198.62:36299
--builtin_ntp_servers=127.10.198.20:45809
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260925 22:45:20.039341 20322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:45:20.030172 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.198.2:39855 (local address 127.10.198.62:36299)
0925 22:45:20.030326 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0925 22:45:20.030330 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:45:20.031538 (+ 1208us) server_negotiation.cc:407] Connection header received
0925 22:45:20.033146 (+ 1608us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:45:20.033150 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:45:20.033241 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:45:20.033332 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:45:20.034500 (+ 1168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:45:20.035202 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:45:20.036848 (+ 1646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:45:20.037099 (+ 251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:45:20.037232 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:45:20.038801 (+ 1569us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:45:20.038804 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:45:20.038806 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 22:45:20.038821 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:45:20.038840 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 22:45:20.038848 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:45:20.038929 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0925 22:45:20.039114 (+ 185us) server_negotiation.cc:299] Negotiation successful
0925 22:45:20.039188 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260925 22:45:20.040491 20603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:45:20.030546 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.198.62:36299 (local address 127.10.198.2:39855)
0925 22:45:20.031348 (+ 802us) negotiation.cc:107] Waiting for socket to connect
0925 22:45:20.031398 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0925 22:45:20.032872 (+ 1474us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:45:20.033532 (+ 660us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:45:20.033540 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:45:20.033640 (+ 100us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:45:20.034283 (+ 643us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:45:20.034308 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:45:20.035388 (+ 1080us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:45:20.035391 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:45:20.036695 (+ 1304us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:45:20.036709 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:45:20.037296 (+ 587us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:45:20.037304 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:45:20.037568 (+ 264us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:45:20.038387 (+ 819us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 22:45:20.038391 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:45:20.038462 (+ 71us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 22:45:20.038580 (+ 118us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:45:20.038942 (+ 362us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:45:20.038954 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:45:20.038960 (+ 6us) client_negotiation.cc:769] Sending connection context
0925 22:45:20.039116 (+ 156us) client_negotiation.cc:240] Negotiation successful
0925 22:45:20.039477 (+ 361us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":345,"thread_start_us":138,"threads_started":1}
I20260925 22:45:20.042212 20601 heartbeater.cc:347] Connected to a master server at 127.10.198.62:36299
I20260925 22:45:20.042501 20601 heartbeater.cc:464] Registering TS with master...
I20260925 22:45:20.043118 20601 heartbeater.cc:511] Master 127.10.198.62:36299 requested a full tablet report, sending...
I20260925 22:45:20.044405 20282 ts_manager.cc:194] Registered new tserver with Master: 879e6ee7b81e417bbae0339eab6b21a1 (127.10.198.2:36195)
I20260925 22:45:20.045028 20282 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.10.198.2:39855
W20260925 22:45:20.161720 20604 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:45:20.162041 20604 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:45:20.162081 20604 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:45:20.166483 20604 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:45:20.166564 20604 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:45:20.166623 20604 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.198.3
I20260925 22:45:20.171751 20604 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.198.20:45809
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790376259650642-11032-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790376259650642-11032-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.198.3:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790376259650642-11032-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.198.3
--webserver_port=0
--tserver_master_addrs=127.10.198.62:36299
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20604
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.198.3
--log_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790376259650642-11032-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:09:16 UTC on ccycloud.quasar-afmhyw.root.comops.site
build id 4673379
I20260925 22:45:20.173398 20604 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:45:20.174882 20604 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:45:20.185320 20612 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:45:20.185320 20609 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:45:20.185333 20610 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:45:20.186060 20604 server_base.cc:1034] running on GCE node
I20260925 22:45:20.186585 20604 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:45:20.187460 20604 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:45:20.190788 20604 hybrid_clock.cc:648] HybridClock initialized: now 1790376320190735 us; error 51 us; skew 500 ppm
I20260925 22:45:20.193193 20604 webserver.cc:460] Webserver started at http://127.10.198.3:45201/ using document root <none> and password file <none>
I20260925 22:45:20.193892 20604 fs_manager.cc:359] Metadata directory not provided
I20260925 22:45:20.193974 20604 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:45:20.194217 20604 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:45:20.196657 20604 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790376259650642-11032-0/minicluster-data/ts-2/data/instance:
uuid: "7448c65887b34b8592d505d61338d0cf"
format_stamp: "Formatted at 2026-09-25 22:45:20 on dist-test-slave-5vdr"
server_key: "7806438f07880b69de7dcbe821e352c9"
server_key_iv: "e591e5692147cb3495d44c61a263d7d2"
server_key_version: "encryptionkey@0"
I20260925 22:45:20.197342 20604 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790376259650642-11032-0/minicluster-data/ts-2/wal/instance:
uuid: "7448c65887b34b8592d505d61338d0cf"
format_stamp: "Formatted at 2026-09-25 22:45:20 on dist-test-slave-5vdr"
server_key: "7806438f07880b69de7dcbe821e352c9"
server_key_iv: "e591e5692147cb3495d44c61a263d7d2"
server_key_version: "encryptionkey@0"
I20260925 22:45:20.201843 20604 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.004s
I20260925 22:45:20.205225 20618 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:45:20.206900 20604 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260925 22:45:20.207051 20604 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790376259650642-11032-0/minicluster-data/ts-2/data,/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790376259650642-11032-0/minicluster-data/ts-2/wal
uuid: "7448c65887b34b8592d505d61338d0cf"
format_stamp: "Formatted at 2026-09-25 22:45:20 on dist-test-slave-5vdr"
server_key: "7806438f07880b69de7dcbe821e352c9"
server_key_iv: "e591e5692147cb3495d44c61a263d7d2"
server_key_version: "encryptionkey@0"
I20260925 22:45:20.207260 20604 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790376259650642-11032-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790376259650642-11032-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790376259650642-11032-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:45:20.230050 20604 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:45:20.230917 20604 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:45:20.231168 20604 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:45:20.231881 20604 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 22:45:20.233004 20604 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:45:20.233089 20604 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:45:20.233163 20604 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:45:20.233208 20604 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 22:45:20.243237 20604 rpc_server.cc:307] RPC server started. Bound to: 127.10.198.3:46785
I20260925 22:45:20.243260 20731 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.198.3:46785 every 8 connection(s)
I20260925 22:45:20.244429 20604 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790376259650642-11032-0/minicluster-data/ts-2/data/info.pb
I20260925 22:45:20.245250 11032 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrWXm8X/build/debug/bin/kudu as pid 20604
I20260925 22:45:20.245400 11032 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790376259650642-11032-0/minicluster-data/ts-2/wal/instance
I20260925 22:45:20.245739 11032 external_mini_cluster.cc:1428] Setting key 522c69a52da22143f457e1c20bc978e3
I20260925 22:45:20.256119 20322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:45:20.247261 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.198.3:40231 (local address 127.10.198.62:36299)
0925 22:45:20.247438 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0925 22:45:20.247445 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 22:45:20.249029 (+ 1584us) server_negotiation.cc:407] Connection header received
0925 22:45:20.250512 (+ 1483us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:45:20.250515 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:45:20.250573 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:45:20.250669 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:45:20.251739 (+ 1070us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:45:20.252409 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:45:20.253794 (+ 1385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:45:20.254049 (+ 255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:45:20.254154 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:45:20.255506 (+ 1352us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:45:20.255510 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:45:20.255512 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 22:45:20.255527 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:45:20.255550 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 22:45:20.255558 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:45:20.255632 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0925 22:45:20.255920 (+ 288us) server_negotiation.cc:299] Negotiation successful
0925 22:45:20.255991 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260925 22:45:20.257161 20734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:45:20.247977 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.198.62:36299 (local address 127.10.198.3:40231)
0925 22:45:20.248862 (+ 885us) negotiation.cc:107] Waiting for socket to connect
0925 22:45:20.248912 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0925 22:45:20.250237 (+ 1325us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:45:20.250860 (+ 623us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:45:20.250869 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:45:20.250967 (+ 98us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:45:20.251549 (+ 582us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:45:20.251563 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:45:20.252570 (+ 1007us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:45:20.252573 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:45:20.253609 (+ 1036us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:45:20.253621 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:45:20.254223 (+ 602us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:45:20.254226 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:45:20.254429 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:45:20.255125 (+ 696us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 22:45:20.255126 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:45:20.255175 (+ 49us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 22:45:20.255256 (+ 81us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:45:20.255697 (+ 441us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:45:20.255706 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:45:20.255711 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 22:45:20.255922 (+ 211us) client_negotiation.cc:240] Negotiation successful
0925 22:45:20.256222 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":397,"thread_start_us":195,"threads_started":1}
I20260925 22:45:20.258556 20732 heartbeater.cc:347] Connected to a master server at 127.10.198.62:36299
I20260925 22:45:20.258854 20732 heartbeater.cc:464] Registering TS with master...
I20260925 22:45:20.259562 20732 heartbeater.cc:511] Master 127.10.198.62:36299 requested a full tablet report, sending...
I20260925 22:45:20.261258 20282 ts_manager.cc:194] Registered new tserver with Master: 7448c65887b34b8592d505d61338d0cf (127.10.198.3:46785)
I20260925 22:45:20.261929 20282 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.10.198.3:40231
I20260925 22:45:20.273500 11032 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260925 22:45:20.276973 20322 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:45:20.275763 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37574 (local address 127.10.198.62:36299)
0925 22:45:20.275972 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0925 22:45:20.275979 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 22:45:20.276097 (+ 118us) server_negotiation.cc:407] Connection header received
0925 22:45:20.276345 (+ 248us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:45:20.276356 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:45:20.276405 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:45:20.276515 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:45:20.276804 (+ 289us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37574: BlockingRecv error: recv got EOF from 127.0.0.1:37574 (error 108)
Metrics: {"server-negotiator.queue_time_us":109}
I20260925 22:45:20.277163 11032 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrWXm8X/build/debug/bin/kudu with pid 20342
I20260925 22:45:20.283879 11032 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrWXm8X/build/debug/bin/kudu with pid 20473
I20260925 22:45:20.290132 11032 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrWXm8X/build/debug/bin/kudu with pid 20604
I20260925 22:45:20.295995 11032 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrWXm8X/build/debug/bin/kudu with pid 20251
2026-09-25T22:45:20Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (1010 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-09-25T22:45:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T22:45:20Z Disabled control of system clock
I20260925 22:45:20.327809 11032 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrWXm8X/build/debug/bin/kudu
/tmp/dist-test-taskrWXm8X/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790376259650642-11032-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790376259650642-11032-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790376259650642-11032-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790376259650642-11032-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.198.62:45219
--webserver_interface=127.10.198.62
--webserver_port=0
--builtin_ntp_servers=127.10.198.20:43511
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.198.62:45219
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260925 22:45:20.449303 20743 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:45:20.449640 20743 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:45:20.449681 20743 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:45:20.454041 20743 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 22:45:20.454118 20743 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:45:20.454133 20743 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:45:20.454151 20743 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 22:45:20.454166 20743 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 22:45:20.459461 20743 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.198.20:43511
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790376259650642-11032-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790376259650642-11032-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.198.62:45219
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.198.62:45219
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790376259650642-11032-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.198.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20743
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790376259650642-11032-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:09:16 UTC on ccycloud.quasar-afmhyw.root.comops.site
build id 4673379
I20260925 22:45:20.460749 20743 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:45:20.462471 20743 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:45:20.472255 20748 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:45:20.472370 20743 server_base.cc:1034] running on GCE node
W20260925 22:45:20.472255 20751 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:45:20.472257 20749 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:45:20.473237 20743 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:45:20.474586 20743 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:45:20.475792 20743 hybrid_clock.cc:648] HybridClock initialized: now 1790376320475777 us; error 52 us; skew 500 ppm
I20260925 22:45:20.478147 20743 webserver.cc:460] Webserver started at http://127.10.198.62:36911/ using document root <none> and password file <none>
I20260925 22:45:20.478888 20743 fs_manager.cc:359] Metadata directory not provided
I20260925 22:45:20.478999 20743 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:45:20.479261 20743 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:45:20.481761 20743 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790376259650642-11032-0/minicluster-data/master-0/data/instance:
uuid: "6d8122b5479f4188b541dfd4b05163fa"
format_stamp: "Formatted at 2026-09-25 22:45:20 on dist-test-slave-5vdr"
server_key: "0732f16233f42cd23b726d77941deaf4"
server_key_iv: "f6375f2d66764800094c0f470c0359c0"
server_key_version: "encryptionkey@0"
I20260925 22:45:20.482434 20743 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790376259650642-11032-0/minicluster-data/master-0/wal/instance:
uuid: "6d8122b5479f4188b541dfd4b05163fa"
format_stamp: "Formatted at 2026-09-25 22:45:20 on dist-test-slave-5vdr"
server_key: "0732f16233f42cd23b726d77941deaf4"
server_key_iv: "f6375f2d66764800094c0f470c0359c0"
server_key_version: "encryptionkey@0"
I20260925 22:45:20.486913 20743 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.005s
I20260925 22:45:20.490028 20757 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:45:20.491596 20743 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260925 22:45:20.491813 20743 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790376259650642-11032-0/minicluster-data/master-0/data,/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790376259650642-11032-0/minicluster-data/master-0/wal
uuid: "6d8122b5479f4188b541dfd4b05163fa"
format_stamp: "Formatted at 2026-09-25 22:45:20 on dist-test-slave-5vdr"
server_key: "0732f16233f42cd23b726d77941deaf4"
server_key_iv: "f6375f2d66764800094c0f470c0359c0"
server_key_version: "encryptionkey@0"
I20260925 22:45:20.492055 20743 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790376259650642-11032-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790376259650642-11032-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790376259650642-11032-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:45:20.497282 20743 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:45:20.498039 20743 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:45:20.498292 20743 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:45:20.506049 20743 rpc_server.cc:307] RPC server started. Bound to: 127.10.198.62:45219
I20260925 22:45:20.506057 20809 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.198.62:45219 every 8 connection(s)
I20260925 22:45:20.507149 20743 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790376259650642-11032-0/minicluster-data/master-0/data/info.pb
I20260925 22:45:20.511338 20810 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:45:20.515228 11032 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrWXm8X/build/debug/bin/kudu as pid 20743
I20260925 22:45:20.515345 11032 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790376259650642-11032-0/minicluster-data/master-0/wal/instance
I20260925 22:45:20.515672 11032 external_mini_cluster.cc:1428] Setting key 2d18db4819de06f81158475dbe37c0de
I20260925 22:45:20.519170 20810 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6d8122b5479f4188b541dfd4b05163fa: Bootstrap starting.
I20260925 22:45:20.522321 20810 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6d8122b5479f4188b541dfd4b05163fa: Neither blocks nor log segments found. Creating new log.
I20260925 22:45:20.522696 20813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:45:20.517381 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47054 (local address 127.10.198.62:45219)
0925 22:45:20.518165 (+ 784us) server_negotiation.cc:206] Beginning negotiation
0925 22:45:20.518188 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0925 22:45:20.518234 (+ 46us) server_negotiation.cc:407] Connection header received
0925 22:45:20.518943 (+ 709us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:45:20.518963 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:45:20.519430 (+ 467us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:45:20.519772 (+ 342us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:45:20.520772 (+ 1000us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:45:20.520777 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:45:20.520791 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 22:45:20.520802 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:45:20.520863 (+ 61us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 22:45:20.520990 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:45:20.521344 (+ 354us) server_negotiation.cc:1035] Waiting for connection context
0925 22:45:20.521542 (+ 198us) server_negotiation.cc:299] Negotiation successful
0925 22:45:20.521821 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":338,"thread_start_us":177,"threads_started":1}
I20260925 22:45:20.523460 20810 log.cc:824] T 00000000000000000000000000000000 P 6d8122b5479f4188b541dfd4b05163fa: Log is configured to *not* fsync() on all Append() calls
I20260925 22:45:20.526017 20810 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6d8122b5479f4188b541dfd4b05163fa: No bootstrap required, opened a new log
I20260925 22:45:20.530321 20810 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6d8122b5479f4188b541dfd4b05163fa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d8122b5479f4188b541dfd4b05163fa" member_type: VOTER last_known_addr { host: "127.10.198.62" port: 45219 } }
I20260925 22:45:20.530622 20810 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6d8122b5479f4188b541dfd4b05163fa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:45:20.530669 20810 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6d8122b5479f4188b541dfd4b05163fa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6d8122b5479f4188b541dfd4b05163fa, State: Initialized, Role: FOLLOWER
I20260925 22:45:20.531177 20810 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6d8122b5479f4188b541dfd4b05163fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d8122b5479f4188b541dfd4b05163fa" member_type: VOTER last_known_addr { host: "127.10.198.62" port: 45219 } }
I20260925 22:45:20.531319 20810 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6d8122b5479f4188b541dfd4b05163fa [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 22:45:20.531366 20810 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6d8122b5479f4188b541dfd4b05163fa [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 22:45:20.531453 20810 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6d8122b5479f4188b541dfd4b05163fa [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:45:20.532606 20810 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6d8122b5479f4188b541dfd4b05163fa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d8122b5479f4188b541dfd4b05163fa" member_type: VOTER last_known_addr { host: "127.10.198.62" port: 45219 } }
I20260925 22:45:20.533002 20810 leader_election.cc:302] T 00000000000000000000000000000000 P 6d8122b5479f4188b541dfd4b05163fa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6d8122b5479f4188b541dfd4b05163fa; no voters:
I20260925 22:45:20.533326 20810 leader_election.cc:288] T 00000000000000000000000000000000 P 6d8122b5479f4188b541dfd4b05163fa [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 22:45:20.533475 20815 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6d8122b5479f4188b541dfd4b05163fa [term 1 FOLLOWER]: Leader election won for term 1
I20260925 22:45:20.533766 20815 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6d8122b5479f4188b541dfd4b05163fa [term 1 LEADER]: Becoming Leader. State: Replica: 6d8122b5479f4188b541dfd4b05163fa, State: Running, Role: LEADER
I20260925 22:45:20.534188 20815 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6d8122b5479f4188b541dfd4b05163fa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d8122b5479f4188b541dfd4b05163fa" member_type: VOTER last_known_addr { host: "127.10.198.62" port: 45219 } }
I20260925 22:45:20.535161 20810 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6d8122b5479f4188b541dfd4b05163fa [sys.catalog]: configured and running, proceeding with master startup.
I20260925 22:45:20.536360 20816 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6d8122b5479f4188b541dfd4b05163fa [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6d8122b5479f4188b541dfd4b05163fa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d8122b5479f4188b541dfd4b05163fa" member_type: VOTER last_known_addr { host: "127.10.198.62" port: 45219 } } }
I20260925 22:45:20.536360 20817 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6d8122b5479f4188b541dfd4b05163fa [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6d8122b5479f4188b541dfd4b05163fa. Latest consensus state: current_term: 1 leader_uuid: "6d8122b5479f4188b541dfd4b05163fa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d8122b5479f4188b541dfd4b05163fa" member_type: VOTER last_known_addr { host: "127.10.198.62" port: 45219 } } }
I20260925 22:45:20.536561 20817 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6d8122b5479f4188b541dfd4b05163fa [sys.catalog]: This master's current role is: LEADER
I20260925 22:45:20.536561 20816 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6d8122b5479f4188b541dfd4b05163fa [sys.catalog]: This master's current role is: LEADER
I20260925 22:45:20.537070 20824 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 22:45:20.540627 20824 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 22:45:20.547686 20824 catalog_manager.cc:1347] Generated new cluster ID: 6834ef3d53b94bd5be884aeddb44350e
I20260925 22:45:20.547816 20824 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 22:45:20.578763 20824 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 22:45:20.579137 20824 catalog_manager.cc:1504] Loading token signing keys...
I20260925 22:45:20.594043 20824 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6d8122b5479f4188b541dfd4b05163fa: Generated new TSK 0
I20260925 22:45:20.594940 20824 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 22:45:20.604476 11032 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrWXm8X/build/debug/bin/kudu
/tmp/dist-test-taskrWXm8X/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790376259650642-11032-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790376259650642-11032-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790376259650642-11032-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790376259650642-11032-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.198.1:0
--local_ip_for_outbound_sockets=127.10.198.1
--webserver_interface=127.10.198.1
--webserver_port=0
--tserver_master_addrs=127.10.198.62:45219
--builtin_ntp_servers=127.10.198.20:43511
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260925 22:45:20.729720 20834 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:45:20.730059 20834 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:45:20.730103 20834 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:45:20.734661 20834 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:45:20.734746 20834 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:45:20.734810 20834 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.198.1
I20260925 22:45:20.740470 20834 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.198.20:43511
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790376259650642-11032-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790376259650642-11032-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.198.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790376259650642-11032-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.198.1
--webserver_port=0
--tserver_master_addrs=127.10.198.62:45219
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20834
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.198.1
--log_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790376259650642-11032-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:09:16 UTC on ccycloud.quasar-afmhyw.root.comops.site
build id 4673379
I20260925 22:45:20.741791 20834 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:45:20.743286 20834 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:45:20.754316 20839 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:45:20.754305 20840 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:45:20.754311 20842 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:45:20.754666 20834 server_base.cc:1034] running on GCE node
I20260925 22:45:20.755208 20834 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:45:20.756052 20834 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:45:20.757282 20834 hybrid_clock.cc:648] HybridClock initialized: now 1790376320757274 us; error 50 us; skew 500 ppm
I20260925 22:45:20.759583 20834 webserver.cc:460] Webserver started at http://127.10.198.1:34765/ using document root <none> and password file <none>
I20260925 22:45:20.760367 20834 fs_manager.cc:359] Metadata directory not provided
I20260925 22:45:20.760485 20834 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:45:20.760730 20834 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:45:20.763134 20834 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790376259650642-11032-0/minicluster-data/ts-0/data/instance:
uuid: "672a4990f26d45b8b6d71d9ff836cdb4"
format_stamp: "Formatted at 2026-09-25 22:45:20 on dist-test-slave-5vdr"
server_key: "9bc26cd7015f37693fdc3dfdde9d9c15"
server_key_iv: "46cab4c806a04b204e2b309400b0d24e"
server_key_version: "encryptionkey@0"
I20260925 22:45:20.763895 20834 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790376259650642-11032-0/minicluster-data/ts-0/wal/instance:
uuid: "672a4990f26d45b8b6d71d9ff836cdb4"
format_stamp: "Formatted at 2026-09-25 22:45:20 on dist-test-slave-5vdr"
server_key: "9bc26cd7015f37693fdc3dfdde9d9c15"
server_key_iv: "46cab4c806a04b204e2b309400b0d24e"
server_key_version: "encryptionkey@0"
I20260925 22:45:20.768631 20834 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260925 22:45:20.772706 20848 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:45:20.774523 20834 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260925 22:45:20.774768 20834 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790376259650642-11032-0/minicluster-data/ts-0/data,/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790376259650642-11032-0/minicluster-data/ts-0/wal
uuid: "672a4990f26d45b8b6d71d9ff836cdb4"
format_stamp: "Formatted at 2026-09-25 22:45:20 on dist-test-slave-5vdr"
server_key: "9bc26cd7015f37693fdc3dfdde9d9c15"
server_key_iv: "46cab4c806a04b204e2b309400b0d24e"
server_key_version: "encryptionkey@0"
I20260925 22:45:20.774942 20834 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790376259650642-11032-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790376259650642-11032-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790376259650642-11032-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:45:20.780436 20834 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:45:20.781260 20834 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:45:20.781527 20834 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:45:20.782235 20834 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 22:45:20.783429 20834 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:45:20.783515 20834 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:45:20.783595 20834 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:45:20.783639 20834 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 22:45:20.794021 20834 rpc_server.cc:307] RPC server started. Bound to: 127.10.198.1:42921
I20260925 22:45:20.794077 20961 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.198.1:42921 every 8 connection(s)
I20260925 22:45:20.795127 20834 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790376259650642-11032-0/minicluster-data/ts-0/data/info.pb
I20260925 22:45:20.802973 20813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:45:20.797626 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.198.1:36927 (local address 127.10.198.62:45219)
0925 22:45:20.797820 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0925 22:45:20.797824 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:45:20.799144 (+ 1320us) server_negotiation.cc:407] Connection header received
0925 22:45:20.800878 (+ 1734us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:45:20.800881 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:45:20.800928 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:45:20.801021 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:45:20.802384 (+ 1363us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:45:20.802388 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:45:20.802390 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 22:45:20.802405 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:45:20.802425 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 22:45:20.802433 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:45:20.802523 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0925 22:45:20.802792 (+ 269us) server_negotiation.cc:299] Negotiation successful
0925 22:45:20.802849 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":97}
I20260925 22:45:20.802966 11032 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrWXm8X/build/debug/bin/kudu as pid 20834
I20260925 22:45:20.803125 11032 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790376259650642-11032-0/minicluster-data/ts-0/wal/instance
I20260925 22:45:20.803443 11032 external_mini_cluster.cc:1428] Setting key b1e846fd2b751d4315f617d7f4b7b63f
I20260925 22:45:20.804311 20964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:45:20.798062 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.198.62:45219 (local address 127.10.198.1:36927)
0925 22:45:20.798905 (+ 843us) negotiation.cc:107] Waiting for socket to connect
0925 22:45:20.798960 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0925 22:45:20.800580 (+ 1620us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:45:20.801287 (+ 707us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:45:20.801296 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:45:20.801403 (+ 107us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:45:20.802047 (+ 644us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 22:45:20.802048 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:45:20.802097 (+ 49us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 22:45:20.802181 (+ 84us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:45:20.802519 (+ 338us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:45:20.802525 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:45:20.802530 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 22:45:20.802777 (+ 247us) client_negotiation.cc:240] Negotiation successful
0925 22:45:20.803091 (+ 314us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":372,"thread_start_us":159,"threads_started":1}
I20260925 22:45:20.806188 20962 heartbeater.cc:347] Connected to a master server at 127.10.198.62:45219
I20260925 22:45:20.806495 20962 heartbeater.cc:464] Registering TS with master...
I20260925 22:45:20.806538 11032 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrWXm8X/build/debug/bin/kudu
/tmp/dist-test-taskrWXm8X/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790376259650642-11032-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790376259650642-11032-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790376259650642-11032-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790376259650642-11032-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.198.2:0
--local_ip_for_outbound_sockets=127.10.198.2
--webserver_interface=127.10.198.2
--webserver_port=0
--tserver_master_addrs=127.10.198.62:45219
--builtin_ntp_servers=127.10.198.20:43511
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260925 22:45:20.807027 20962 heartbeater.cc:511] Master 127.10.198.62:45219 requested a full tablet report, sending...
I20260925 22:45:20.808876 20774 ts_manager.cc:194] Registered new tserver with Master: 672a4990f26d45b8b6d71d9ff836cdb4 (127.10.198.1:42921)
W20260925 22:45:20.930032 20965 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:45:20.930408 20965 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:45:20.930485 20965 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:45:20.935335 20965 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:45:20.935453 20965 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:45:20.935561 20965 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.198.2
I20260925 22:45:20.941810 20965 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.198.20:43511
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790376259650642-11032-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790376259650642-11032-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.198.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790376259650642-11032-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.198.2
--webserver_port=0
--tserver_master_addrs=127.10.198.62:45219
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20965
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.198.2
--log_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790376259650642-11032-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:09:16 UTC on ccycloud.quasar-afmhyw.root.comops.site
build id 4673379
I20260925 22:45:20.943066 20965 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:45:20.944702 20965 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:45:20.955596 20971 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:45:20.955659 20973 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:45:20.955871 20965 server_base.cc:1034] running on GCE node
W20260925 22:45:20.955637 20970 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:45:20.956455 20965 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:45:20.957315 20965 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:45:20.958545 20965 hybrid_clock.cc:648] HybridClock initialized: now 1790376320958528 us; error 37 us; skew 500 ppm
I20260925 22:45:20.961042 20965 webserver.cc:460] Webserver started at http://127.10.198.2:38669/ using document root <none> and password file <none>
I20260925 22:45:20.961822 20965 fs_manager.cc:359] Metadata directory not provided
I20260925 22:45:20.961942 20965 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:45:20.962193 20965 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:45:20.964692 20965 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790376259650642-11032-0/minicluster-data/ts-1/data/instance:
uuid: "15d93ef3cc0f42efbc93dc650178eeb6"
format_stamp: "Formatted at 2026-09-25 22:45:20 on dist-test-slave-5vdr"
server_key: "1bfdab6e44c76b966823ff15bf801bbc"
server_key_iv: "32dea3d72feef5312d9290ff74e19598"
server_key_version: "encryptionkey@0"
I20260925 22:45:20.965370 20965 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790376259650642-11032-0/minicluster-data/ts-1/wal/instance:
uuid: "15d93ef3cc0f42efbc93dc650178eeb6"
format_stamp: "Formatted at 2026-09-25 22:45:20 on dist-test-slave-5vdr"
server_key: "1bfdab6e44c76b966823ff15bf801bbc"
server_key_iv: "32dea3d72feef5312d9290ff74e19598"
server_key_version: "encryptionkey@0"
I20260925 22:45:20.969760 20965 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.004s
I20260925 22:45:20.973048 20979 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:45:20.974704 20965 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260925 22:45:20.974920 20965 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790376259650642-11032-0/minicluster-data/ts-1/data,/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790376259650642-11032-0/minicluster-data/ts-1/wal
uuid: "15d93ef3cc0f42efbc93dc650178eeb6"
format_stamp: "Formatted at 2026-09-25 22:45:20 on dist-test-slave-5vdr"
server_key: "1bfdab6e44c76b966823ff15bf801bbc"
server_key_iv: "32dea3d72feef5312d9290ff74e19598"
server_key_version: "encryptionkey@0"
I20260925 22:45:20.975082 20965 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790376259650642-11032-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790376259650642-11032-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790376259650642-11032-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:45:20.980756 20965 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:45:20.981526 20965 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:45:20.981760 20965 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:45:20.982406 20965 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 22:45:20.983532 20965 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:45:20.983620 20965 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:45:20.983695 20965 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:45:20.983738 20965 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 22:45:20.994258 20965 rpc_server.cc:307] RPC server started. Bound to: 127.10.198.2:46587
I20260925 22:45:20.994282 21092 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.198.2:46587 every 8 connection(s)
I20260925 22:45:20.995340 20965 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790376259650642-11032-0/minicluster-data/ts-1/data/info.pb
I20260925 22:45:21.002509 20813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:45:20.997586 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.198.2:34641 (local address 127.10.198.62:45219)
0925 22:45:20.997767 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0925 22:45:20.997771 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:45:20.998949 (+ 1178us) server_negotiation.cc:407] Connection header received
0925 22:45:21.000474 (+ 1525us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:45:21.000480 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:45:21.000531 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:45:21.000599 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:45:21.001962 (+ 1363us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:45:21.001972 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:45:21.001974 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 22:45:21.001988 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:45:21.002010 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 22:45:21.002019 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:45:21.002132 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0925 22:45:21.002299 (+ 167us) server_negotiation.cc:299] Negotiation successful
0925 22:45:21.002352 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
I20260925 22:45:21.003530 21095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:45:20.997985 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.198.62:45219 (local address 127.10.198.2:34641)
0925 22:45:20.998772 (+ 787us) negotiation.cc:107] Waiting for socket to connect
0925 22:45:20.998820 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0925 22:45:21.000218 (+ 1398us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:45:21.000832 (+ 614us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:45:21.000840 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:45:21.000947 (+ 107us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:45:21.001602 (+ 655us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 22:45:21.001604 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:45:21.001652 (+ 48us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 22:45:21.001767 (+ 115us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:45:21.002119 (+ 352us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:45:21.002127 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:45:21.002143 (+ 16us) client_negotiation.cc:769] Sending connection context
0925 22:45:21.002355 (+ 212us) client_negotiation.cc:240] Negotiation successful
0925 22:45:21.002656 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":303,"thread_start_us":135,"threads_started":1}
I20260925 22:45:21.004151 11032 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrWXm8X/build/debug/bin/kudu as pid 20965
I20260925 22:45:21.004321 11032 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790376259650642-11032-0/minicluster-data/ts-1/wal/instance
I20260925 22:45:21.004698 11032 external_mini_cluster.cc:1428] Setting key 31d781446eed41bc4209d53f95aa3196
I20260925 22:45:21.005044 21093 heartbeater.cc:347] Connected to a master server at 127.10.198.62:45219
I20260925 22:45:21.005360 21093 heartbeater.cc:464] Registering TS with master...
I20260925 22:45:21.005887 21093 heartbeater.cc:511] Master 127.10.198.62:45219 requested a full tablet report, sending...
I20260925 22:45:21.006865 11032 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrWXm8X/build/debug/bin/kudu
/tmp/dist-test-taskrWXm8X/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790376259650642-11032-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790376259650642-11032-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790376259650642-11032-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790376259650642-11032-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.198.3:0
--local_ip_for_outbound_sockets=127.10.198.3
--webserver_interface=127.10.198.3
--webserver_port=0
--tserver_master_addrs=127.10.198.62:45219
--builtin_ntp_servers=127.10.198.20:43511
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260925 22:45:21.007082 20774 ts_manager.cc:194] Registered new tserver with Master: 15d93ef3cc0f42efbc93dc650178eeb6 (127.10.198.2:46587)
W20260925 22:45:21.131222 21096 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:45:21.131592 21096 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:45:21.131683 21096 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:45:21.136709 21096 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:45:21.136809 21096 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:45:21.136907 21096 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.198.3
I20260925 22:45:21.142493 21096 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.198.20:43511
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790376259650642-11032-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790376259650642-11032-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.198.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790376259650642-11032-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.198.3
--webserver_port=0
--tserver_master_addrs=127.10.198.62:45219
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21096
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.198.3
--log_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790376259650642-11032-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:09:16 UTC on ccycloud.quasar-afmhyw.root.comops.site
build id 4673379
I20260925 22:45:21.143829 21096 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:45:21.145386 21096 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:45:21.156160 21102 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:45:21.156190 21104 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:45:21.156181 21101 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:45:21.156549 21096 server_base.cc:1034] running on GCE node
I20260925 22:45:21.157020 21096 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:45:21.157776 21096 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:45:21.159013 21096 hybrid_clock.cc:648] HybridClock initialized: now 1790376321158983 us; error 49 us; skew 500 ppm
I20260925 22:45:21.161521 21096 webserver.cc:460] Webserver started at http://127.10.198.3:38289/ using document root <none> and password file <none>
I20260925 22:45:21.162205 21096 fs_manager.cc:359] Metadata directory not provided
I20260925 22:45:21.162319 21096 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:45:21.162585 21096 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:45:21.165145 21096 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790376259650642-11032-0/minicluster-data/ts-2/data/instance:
uuid: "b912e737d08d46e79348b86cae375fd4"
format_stamp: "Formatted at 2026-09-25 22:45:21 on dist-test-slave-5vdr"
server_key: "f7b132918d036e8984111fd83e757ad1"
server_key_iv: "0a1ec250450a6cf5e2b8e23bf53d7fc9"
server_key_version: "encryptionkey@0"
I20260925 22:45:21.165794 21096 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790376259650642-11032-0/minicluster-data/ts-2/wal/instance:
uuid: "b912e737d08d46e79348b86cae375fd4"
format_stamp: "Formatted at 2026-09-25 22:45:21 on dist-test-slave-5vdr"
server_key: "f7b132918d036e8984111fd83e757ad1"
server_key_iv: "0a1ec250450a6cf5e2b8e23bf53d7fc9"
server_key_version: "encryptionkey@0"
I20260925 22:45:21.170321 21096 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.003s
I20260925 22:45:21.173465 21110 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:45:21.175000 21096 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260925 22:45:21.175215 21096 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790376259650642-11032-0/minicluster-data/ts-2/data,/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790376259650642-11032-0/minicluster-data/ts-2/wal
uuid: "b912e737d08d46e79348b86cae375fd4"
format_stamp: "Formatted at 2026-09-25 22:45:21 on dist-test-slave-5vdr"
server_key: "f7b132918d036e8984111fd83e757ad1"
server_key_iv: "0a1ec250450a6cf5e2b8e23bf53d7fc9"
server_key_version: "encryptionkey@0"
I20260925 22:45:21.175377 21096 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790376259650642-11032-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790376259650642-11032-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790376259650642-11032-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:45:21.180938 21096 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:45:21.181711 21096 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:45:21.181938 21096 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:45:21.182561 21096 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 22:45:21.183626 21096 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:45:21.183706 21096 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:45:21.183785 21096 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:45:21.183840 21096 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 22:45:21.193754 21096 rpc_server.cc:307] RPC server started. Bound to: 127.10.198.3:40649
I20260925 22:45:21.193778 21223 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.198.3:40649 every 8 connection(s)
I20260925 22:45:21.194836 21096 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790376259650642-11032-0/minicluster-data/ts-2/data/info.pb
I20260925 22:45:21.202935 20813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:45:21.197087 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.198.3:48763 (local address 127.10.198.62:45219)
0925 22:45:21.197256 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0925 22:45:21.197261 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:45:21.198644 (+ 1383us) server_negotiation.cc:407] Connection header received
0925 22:45:21.200192 (+ 1548us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:45:21.200196 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:45:21.200260 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:45:21.200336 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:45:21.202165 (+ 1829us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:45:21.202169 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:45:21.202171 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 22:45:21.202186 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:45:21.202208 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 22:45:21.202217 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:45:21.202338 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0925 22:45:21.202735 (+ 397us) server_negotiation.cc:299] Negotiation successful
0925 22:45:21.202787 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260925 22:45:21.204142 21226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:45:21.197645 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.198.62:45219 (local address 127.10.198.3:48763)
0925 22:45:21.198443 (+ 798us) negotiation.cc:107] Waiting for socket to connect
0925 22:45:21.198495 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0925 22:45:21.199805 (+ 1310us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:45:21.200787 (+ 982us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:45:21.200800 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:45:21.200947 (+ 147us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:45:21.201766 (+ 819us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 22:45:21.201768 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:45:21.201819 (+ 51us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 22:45:21.201920 (+ 101us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:45:21.202381 (+ 461us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:45:21.202395 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:45:21.202404 (+ 9us) client_negotiation.cc:769] Sending connection context
0925 22:45:21.202741 (+ 337us) client_negotiation.cc:240] Negotiation successful
0925 22:45:21.203102 (+ 361us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":322,"thread_start_us":140,"threads_started":1}
I20260925 22:45:21.204583 11032 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrWXm8X/build/debug/bin/kudu as pid 21096
I20260925 22:45:21.204674 11032 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790376259650642-11032-0/minicluster-data/ts-2/wal/instance
I20260925 22:45:21.205022 11032 external_mini_cluster.cc:1428] Setting key dd9b18bba72944a3ae3b35f2145f50fb
I20260925 22:45:21.205737 21224 heartbeater.cc:347] Connected to a master server at 127.10.198.62:45219
I20260925 22:45:21.206073 21224 heartbeater.cc:464] Registering TS with master...
I20260925 22:45:21.206597 21224 heartbeater.cc:511] Master 127.10.198.62:45219 requested a full tablet report, sending...
I20260925 22:45:21.207721 20774 ts_manager.cc:194] Registered new tserver with Master: b912e737d08d46e79348b86cae375fd4 (127.10.198.3:40649)
I20260925 22:45:21.219816 11032 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260925 22:45:21.223284 20813 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:45:21.222058 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43036 (local address 127.10.198.62:45219)
0925 22:45:21.222243 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0925 22:45:21.222247 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:45:21.222399 (+ 152us) server_negotiation.cc:407] Connection header received
0925 22:45:21.222609 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:45:21.222612 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:45:21.222656 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:45:21.222758 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:45:21.223122 (+ 364us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43036: BlockingRecv error: recv got EOF from 127.0.0.1:43036 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
I20260925 22:45:21.223523 11032 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrWXm8X/build/debug/bin/kudu with pid 20834
I20260925 22:45:21.229606 11032 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrWXm8X/build/debug/bin/kudu with pid 20965
I20260925 22:45:21.235642 11032 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrWXm8X/build/debug/bin/kudu with pid 21096
I20260925 22:45:21.241760 11032 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrWXm8X/build/debug/bin/kudu with pid 20743
2026-09-25T22:45:21Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (945 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790376259650642-11032-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 22:45:21 dist-test-slave-5vdr krb5kdc[21234](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 22:45:21 dist-test-slave-5vdr krb5kdc[21234](info): set up 2 sockets
Sep 25 22:45:21 dist-test-slave-5vdr krb5kdc[21234](info): commencing operation
krb5kdc: starting...
W20260925 22:45:22.317132 11032 mini_kdc.cc:121] Time spent starting KDC: real 1.053s user 0.001s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790376259650642-11032-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790376259650642-11032-0/krb5kdc/test-user.keytab.
Sep 25 22:45:22 dist-test-slave-5vdr krb5kdc[21234](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376322, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T22:45:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T22:45:22Z Disabled control of system clock
WARNING: no policy specified for kudu/127.10.198.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.198.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.198.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790376259650642-11032-0/krb5kdc/kudu_127.10.198.62.keytab.
Entry for principal kudu/127.10.198.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790376259650642-11032-0/krb5kdc/kudu_127.10.198.62.keytab.
I20260925 22:45:22.451152 11032 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrWXm8X/build/debug/bin/kudu
/tmp/dist-test-taskrWXm8X/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790376259650642-11032-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790376259650642-11032-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790376259650642-11032-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790376259650642-11032-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.198.62:33343
--webserver_interface=127.10.198.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790376259650642-11032-0/krb5kdc/kudu_127.10.198.62.keytab
--principal=kudu/127.10.198.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.198.20:37281
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.198.62:33343
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790376259650642-11032-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790376259650642-11032-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790376259650642-11032-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:45:22.574368 21247 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:45:22.574699 21247 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:45:22.574743 21247 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:45:22.578969 21247 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 22:45:22.579047 21247 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:45:22.579068 21247 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:45:22.579085 21247 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 22:45:22.579121 21247 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 22:45:22.584904 21247 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.198.20:37281
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790376259650642-11032-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790376259650642-11032-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.198.62:33343
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790376259650642-11032-0/krb5kdc/kudu_127.10.198.62.keytab
--principal=kudu/127.10.198.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.198.62:33343
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790376259650642-11032-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.198.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21247
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790376259650642-11032-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:09:16 UTC on ccycloud.quasar-afmhyw.root.comops.site
build id 4673379
I20260925 22:45:22.586076 21247 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:45:22.587128 21247 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:45:22.596843 21255 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:45:22.596948 21253 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:45:22.596832 21252 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:45:22.597247 21247 server_base.cc:1034] running on GCE node
I20260925 22:45:22.597967 21247 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:45:22.599341 21247 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:45:22.600549 21247 hybrid_clock.cc:648] HybridClock initialized: now 1790376322600482 us; error 94 us; skew 500 ppm
Sep 25 22:45:22 dist-test-slave-5vdr krb5kdc[21234](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376322, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.198.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:45:22.604180 21247 init.cc:377] Logged in from keytab as kudu/127.10.198.62@KRBTEST.COM (short username kudu)
I20260925 22:45:22.605253 21247 webserver.cc:460] Webserver started at http://127.10.198.62:37913/ using document root <none> and password file <none>
I20260925 22:45:22.605862 21247 fs_manager.cc:359] Metadata directory not provided
I20260925 22:45:22.605935 21247 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:45:22.606122 21247 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:45:22.608636 21247 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790376259650642-11032-0/minicluster-data/master-0/data/instance:
uuid: "f1830946663e4d1c89a857ffce4de755"
format_stamp: "Formatted at 2026-09-25 22:45:22 on dist-test-slave-5vdr"
server_key: "adc5d2c08adc83f59a1b946f1fe1662f"
server_key_iv: "c260d85444c7ddec67cda658f6c64f46"
server_key_version: "encryptionkey@0"
I20260925 22:45:22.609284 21247 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790376259650642-11032-0/minicluster-data/master-0/wal/instance:
uuid: "f1830946663e4d1c89a857ffce4de755"
format_stamp: "Formatted at 2026-09-25 22:45:22 on dist-test-slave-5vdr"
server_key: "adc5d2c08adc83f59a1b946f1fe1662f"
server_key_iv: "c260d85444c7ddec67cda658f6c64f46"
server_key_version: "encryptionkey@0"
I20260925 22:45:22.613857 21247 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260925 22:45:22.616955 21262 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:45:22.618638 21247 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 22:45:22.618819 21247 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790376259650642-11032-0/minicluster-data/master-0/data,/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790376259650642-11032-0/minicluster-data/master-0/wal
uuid: "f1830946663e4d1c89a857ffce4de755"
format_stamp: "Formatted at 2026-09-25 22:45:22 on dist-test-slave-5vdr"
server_key: "adc5d2c08adc83f59a1b946f1fe1662f"
server_key_iv: "c260d85444c7ddec67cda658f6c64f46"
server_key_version: "encryptionkey@0"
I20260925 22:45:22.618973 21247 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790376259650642-11032-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790376259650642-11032-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790376259650642-11032-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:45:22.635318 21247 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:45:22.639559 21247 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:45:22.639819 21247 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:45:22.647727 21247 rpc_server.cc:307] RPC server started. Bound to: 127.10.198.62:33343
I20260925 22:45:22.648103 21314 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.198.62:33343 every 8 connection(s)
I20260925 22:45:22.648983 21247 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790376259650642-11032-0/minicluster-data/master-0/data/info.pb
I20260925 22:45:22.652247 21315 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:45:22.658735 21315 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f1830946663e4d1c89a857ffce4de755: Bootstrap starting.
I20260925 22:45:22.659241 11032 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrWXm8X/build/debug/bin/kudu as pid 21247
I20260925 22:45:22.659453 11032 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790376259650642-11032-0/minicluster-data/master-0/wal/instance
I20260925 22:45:22.659766 11032 external_mini_cluster.cc:1428] Setting key 87eff8eaa0f6a9dfb031be4535cb4c05
I20260925 22:45:22.661927 21315 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f1830946663e4d1c89a857ffce4de755: Neither blocks nor log segments found. Creating new log.
I20260925 22:45:22.662802 21315 log.cc:824] T 00000000000000000000000000000000 P f1830946663e4d1c89a857ffce4de755: Log is configured to *not* fsync() on all Append() calls
I20260925 22:45:22.665627 21315 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f1830946663e4d1c89a857ffce4de755: No bootstrap required, opened a new log
Sep 25 22:45:22 dist-test-slave-5vdr krb5kdc[21234](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376322, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.198.62@KRBTEST.COM
I20260925 22:45:22.670985 21315 raft_consensus.cc:348] T 00000000000000000000000000000000 P f1830946663e4d1c89a857ffce4de755 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f1830946663e4d1c89a857ffce4de755" member_type: VOTER last_known_addr { host: "127.10.198.62" port: 33343 } }
I20260925 22:45:22.671306 21315 raft_consensus.cc:374] T 00000000000000000000000000000000 P f1830946663e4d1c89a857ffce4de755 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:45:22.671394 21315 raft_consensus.cc:729] T 00000000000000000000000000000000 P f1830946663e4d1c89a857ffce4de755 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1830946663e4d1c89a857ffce4de755, State: Initialized, Role: FOLLOWER
I20260925 22:45:22.672047 21315 consensus_queue.cc:260] T 00000000000000000000000000000000 P f1830946663e4d1c89a857ffce4de755 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f1830946663e4d1c89a857ffce4de755" member_type: VOTER last_known_addr { host: "127.10.198.62" port: 33343 } }
I20260925 22:45:22.672241 21315 raft_consensus.cc:388] T 00000000000000000000000000000000 P f1830946663e4d1c89a857ffce4de755 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 22:45:22.672339 21315 raft_consensus.cc:482] T 00000000000000000000000000000000 P f1830946663e4d1c89a857ffce4de755 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 22:45:22.672487 21315 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f1830946663e4d1c89a857ffce4de755 [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:45:22.673939 21315 raft_consensus.cc:504] T 00000000000000000000000000000000 P f1830946663e4d1c89a857ffce4de755 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f1830946663e4d1c89a857ffce4de755" member_type: VOTER last_known_addr { host: "127.10.198.62" port: 33343 } }
I20260925 22:45:22.674405 21315 leader_election.cc:302] T 00000000000000000000000000000000 P f1830946663e4d1c89a857ffce4de755 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f1830946663e4d1c89a857ffce4de755; no voters:
I20260925 22:45:22.674830 21315 leader_election.cc:288] T 00000000000000000000000000000000 P f1830946663e4d1c89a857ffce4de755 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 22:45:22.674912 21320 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f1830946663e4d1c89a857ffce4de755 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 22:45:22.675113 21320 raft_consensus.cc:686] T 00000000000000000000000000000000 P f1830946663e4d1c89a857ffce4de755 [term 1 LEADER]: Becoming Leader. State: Replica: f1830946663e4d1c89a857ffce4de755, State: Running, Role: LEADER
I20260925 22:45:22.675455 21320 consensus_queue.cc:237] T 00000000000000000000000000000000 P f1830946663e4d1c89a857ffce4de755 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f1830946663e4d1c89a857ffce4de755" member_type: VOTER last_known_addr { host: "127.10.198.62" port: 33343 } }
I20260925 22:45:22.676473 21315 sys_catalog.cc:564] T 00000000000000000000000000000000 P f1830946663e4d1c89a857ffce4de755 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 22:45:22.677560 21322 sys_catalog.cc:455] T 00000000000000000000000000000000 P f1830946663e4d1c89a857ffce4de755 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f1830946663e4d1c89a857ffce4de755. Latest consensus state: current_term: 1 leader_uuid: "f1830946663e4d1c89a857ffce4de755" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f1830946663e4d1c89a857ffce4de755" member_type: VOTER last_known_addr { host: "127.10.198.62" port: 33343 } } }
I20260925 22:45:22.678054 21322 sys_catalog.cc:458] T 00000000000000000000000000000000 P f1830946663e4d1c89a857ffce4de755 [sys.catalog]: This master's current role is: LEADER
I20260925 22:45:22.678262 21318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:45:22.661281 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54950 (local address 127.10.198.62:33343)
0925 22:45:22.661813 (+ 532us) server_negotiation.cc:206] Beginning negotiation
0925 22:45:22.661831 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0925 22:45:22.661877 (+ 46us) server_negotiation.cc:407] Connection header received
0925 22:45:22.662720 (+ 843us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:45:22.662744 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:45:22.663301 (+ 557us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:45:22.663818 (+ 517us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:45:22.664915 (+ 1097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:45:22.666113 (+ 1198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:45:22.667394 (+ 1281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:45:22.668102 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:45:22.668274 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:45:22.671684 (+ 3410us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:45:22.671691 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:45:22.671711 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:45:22.671757 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:45:22.674148 (+ 2391us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:45:22.674748 (+ 600us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:45:22.674758 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:45:22.674770 (+ 12us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:45:22.674946 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:45:22.675411 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:45:22.675416 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:45:22.675419 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:45:22.676048 (+ 629us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:45:22.676317 (+ 269us) server_negotiation.cc:1035] Waiting for connection context
0925 22:45:22.676648 (+ 331us) server_negotiation.cc:299] Negotiation successful
0925 22:45:22.677001 (+ 353us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":368,"thread_start_us":157,"threads_started":1}
I20260925 22:45:22.677949 21321 sys_catalog.cc:455] T 00000000000000000000000000000000 P f1830946663e4d1c89a857ffce4de755 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f1830946663e4d1c89a857ffce4de755" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f1830946663e4d1c89a857ffce4de755" member_type: VOTER last_known_addr { host: "127.10.198.62" port: 33343 } } }
I20260925 22:45:22.678639 21321 sys_catalog.cc:458] T 00000000000000000000000000000000 P f1830946663e4d1c89a857ffce4de755 [sys.catalog]: This master's current role is: LEADER
I20260925 22:45:22.679185 21331 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 22:45:22.682116 21331 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 22:45:22.689037 21331 catalog_manager.cc:1347] Generated new cluster ID: 8fe0bbdabe534c7ba479fadf95e9fa21
I20260925 22:45:22.689141 21331 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 22:45:22.695663 21331 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 22:45:22.696743 21331 catalog_manager.cc:1504] Loading token signing keys...
I20260925 22:45:22.705648 21331 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f1830946663e4d1c89a857ffce4de755: Generated new TSK 0
I20260925 22:45:22.706493 21331 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.10.198.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.198.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.198.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790376259650642-11032-0/krb5kdc/kudu_127.10.198.1.keytab.
Entry for principal kudu/127.10.198.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790376259650642-11032-0/krb5kdc/kudu_127.10.198.1.keytab.
I20260925 22:45:22.740314 11032 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrWXm8X/build/debug/bin/kudu
/tmp/dist-test-taskrWXm8X/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790376259650642-11032-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790376259650642-11032-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790376259650642-11032-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790376259650642-11032-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.198.1:0
--local_ip_for_outbound_sockets=127.10.198.1
--webserver_interface=127.10.198.1
--webserver_port=0
--tserver_master_addrs=127.10.198.62:33343
--keytab_file=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790376259650642-11032-0/krb5kdc/kudu_127.10.198.1.keytab
--principal=kudu/127.10.198.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.198.20:37281
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790376259650642-11032-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790376259650642-11032-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790376259650642-11032-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:45:22.865676 21341 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:45:22.866008 21341 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:45:22.866056 21341 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:45:22.870501 21341 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:45:22.870594 21341 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:45:22.870658 21341 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.198.1
I20260925 22:45:22.876631 21341 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.198.20:37281
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790376259650642-11032-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790376259650642-11032-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790376259650642-11032-0/krb5kdc/kudu_127.10.198.1.keytab
--principal=kudu/127.10.198.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.198.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790376259650642-11032-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.198.1
--webserver_port=0
--tserver_master_addrs=127.10.198.62:33343
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21341
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.198.1
--log_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790376259650642-11032-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:09:16 UTC on ccycloud.quasar-afmhyw.root.comops.site
build id 4673379
I20260925 22:45:22.878162 21341 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:45:22.879360 21341 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:45:22.890522 21349 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:45:22.890585 21347 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:45:22.890707 21341 server_base.cc:1034] running on GCE node
W20260925 22:45:22.890527 21346 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:45:22.891444 21341 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:45:22.892390 21341 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:45:22.893594 21341 hybrid_clock.cc:648] HybridClock initialized: now 1790376322893572 us; error 40 us; skew 500 ppm
Sep 25 22:45:22 dist-test-slave-5vdr krb5kdc[21234](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376322, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.198.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:45:22.896723 21341 init.cc:377] Logged in from keytab as kudu/127.10.198.1@KRBTEST.COM (short username kudu)
I20260925 22:45:22.897850 21341 webserver.cc:460] Webserver started at http://127.10.198.1:34045/ using document root <none> and password file <none>
I20260925 22:45:22.898497 21341 fs_manager.cc:359] Metadata directory not provided
I20260925 22:45:22.898572 21341 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:45:22.898837 21341 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:45:22.901271 21341 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790376259650642-11032-0/minicluster-data/ts-0/data/instance:
uuid: "54d82bc61dea4af89266201d8a1deb47"
format_stamp: "Formatted at 2026-09-25 22:45:22 on dist-test-slave-5vdr"
server_key: "d92a4fc7c6ea72377e4cc038e7686cb4"
server_key_iv: "f96868f272212244f6ea680305abdca4"
server_key_version: "encryptionkey@0"
I20260925 22:45:22.901966 21341 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790376259650642-11032-0/minicluster-data/ts-0/wal/instance:
uuid: "54d82bc61dea4af89266201d8a1deb47"
format_stamp: "Formatted at 2026-09-25 22:45:22 on dist-test-slave-5vdr"
server_key: "d92a4fc7c6ea72377e4cc038e7686cb4"
server_key_iv: "f96868f272212244f6ea680305abdca4"
server_key_version: "encryptionkey@0"
I20260925 22:45:22.906369 21341 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260925 22:45:22.909485 21356 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:45:22.911058 21341 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260925 22:45:22.911264 21341 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790376259650642-11032-0/minicluster-data/ts-0/data,/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790376259650642-11032-0/minicluster-data/ts-0/wal
uuid: "54d82bc61dea4af89266201d8a1deb47"
format_stamp: "Formatted at 2026-09-25 22:45:22 on dist-test-slave-5vdr"
server_key: "d92a4fc7c6ea72377e4cc038e7686cb4"
server_key_iv: "f96868f272212244f6ea680305abdca4"
server_key_version: "encryptionkey@0"
I20260925 22:45:22.911417 21341 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790376259650642-11032-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790376259650642-11032-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790376259650642-11032-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:45:22.927414 21341 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:45:22.931924 21341 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:45:22.932163 21341 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:45:22.932842 21341 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 22:45:22.933792 21341 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:45:22.933845 21341 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:45:22.933923 21341 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:45:22.933954 21341 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 22:45:22.943776 21341 rpc_server.cc:307] RPC server started. Bound to: 127.10.198.1:38947
I20260925 22:45:22.943797 21469 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.198.1:38947 every 8 connection(s)
I20260925 22:45:22.944859 21341 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790376259650642-11032-0/minicluster-data/ts-0/data/info.pb
I20260925 22:45:22.948369 11032 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrWXm8X/build/debug/bin/kudu as pid 21341
I20260925 22:45:22.948534 11032 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790376259650642-11032-0/minicluster-data/ts-0/wal/instance
I20260925 22:45:22.948873 11032 external_mini_cluster.cc:1428] Setting key f30065edecc0581d5466ea12cd42469e
Sep 25 22:45:22 dist-test-slave-5vdr krb5kdc[21234](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376322, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.198.1@KRBTEST.COM for kudu/127.10.198.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.10.198.2@KRBTEST.COM; defaulting to no policy
I20260925 22:45:22.962399 21318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:45:22.947362 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.198.1:39619 (local address 127.10.198.62:33343)
0925 22:45:22.947523 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0925 22:45:22.947527 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:45:22.948828 (+ 1301us) server_negotiation.cc:407] Connection header received
0925 22:45:22.950507 (+ 1679us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:45:22.950510 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:45:22.950566 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:45:22.950659 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:45:22.952429 (+ 1770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:45:22.952977 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:45:22.954444 (+ 1467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:45:22.954876 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:45:22.955027 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:45:22.958198 (+ 3171us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:45:22.958202 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:45:22.958204 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:45:22.958233 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:45:22.959963 (+ 1730us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:45:22.960599 (+ 636us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:45:22.960602 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:45:22.960604 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:45:22.960702 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:45:22.961097 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:45:22.961100 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:45:22.961102 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:45:22.961329 (+ 227us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:45:22.961448 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0925 22:45:22.962053 (+ 605us) server_negotiation.cc:299] Negotiation successful
0925 22:45:22.962213 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.198.2@KRBTEST.COM" created.
I20260925 22:45:22.963461 21472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:45:22.947985 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.198.62:33343 (local address 127.10.198.1:39619)
0925 22:45:22.948606 (+ 621us) negotiation.cc:107] Waiting for socket to connect
0925 22:45:22.948680 (+ 74us) client_negotiation.cc:174] Beginning negotiation
0925 22:45:22.950194 (+ 1514us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:45:22.950894 (+ 700us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:45:22.950903 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:45:22.951448 (+ 545us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:45:22.952178 (+ 730us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:45:22.952218 (+ 40us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:45:22.953140 (+ 922us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:45:22.953143 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:45:22.954259 (+ 1116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:45:22.954272 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:45:22.955060 (+ 788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:45:22.955064 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:45:22.955249 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:45:22.955986 (+ 737us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:45:22.956016 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:45:22.957952 (+ 1936us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:45:22.960104 (+ 2152us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:45:22.960112 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:45:22.960135 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:45:22.960462 (+ 327us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:45:22.960811 (+ 349us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:45:22.960814 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:45:22.960817 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:45:22.960983 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:45:22.961465 (+ 482us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:45:22.961475 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:45:22.961783 (+ 308us) client_negotiation.cc:769] Sending connection context
0925 22:45:22.961995 (+ 212us) client_negotiation.cc:240] Negotiation successful
0925 22:45:22.962367 (+ 372us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":388,"thread_start_us":182,"threads_started":1}
I20260925 22:45:22.965238 21470 heartbeater.cc:347] Connected to a master server at 127.10.198.62:33343
I20260925 22:45:22.965580 21470 heartbeater.cc:464] Registering TS with master...
I20260925 22:45:22.966164 21470 heartbeater.cc:511] Master 127.10.198.62:33343 requested a full tablet report, sending...
I20260925 22:45:22.968036 21279 ts_manager.cc:194] Registered new tserver with Master: 54d82bc61dea4af89266201d8a1deb47 (127.10.198.1:38947)
I20260925 22:45:22.969533 21279 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.198.1@KRBTEST.COM'} at 127.10.198.1:39619
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.198.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790376259650642-11032-0/krb5kdc/kudu_127.10.198.2.keytab.
Entry for principal kudu/127.10.198.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790376259650642-11032-0/krb5kdc/kudu_127.10.198.2.keytab.
I20260925 22:45:22.981287 11032 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrWXm8X/build/debug/bin/kudu
/tmp/dist-test-taskrWXm8X/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790376259650642-11032-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790376259650642-11032-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790376259650642-11032-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790376259650642-11032-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.198.2:0
--local_ip_for_outbound_sockets=127.10.198.2
--webserver_interface=127.10.198.2
--webserver_port=0
--tserver_master_addrs=127.10.198.62:33343
--keytab_file=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790376259650642-11032-0/krb5kdc/kudu_127.10.198.2.keytab
--principal=kudu/127.10.198.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.198.20:37281
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790376259650642-11032-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790376259650642-11032-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790376259650642-11032-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:45:23.102604 21475 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:45:23.102949 21475 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:45:23.102998 21475 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:45:23.107900 21475 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:45:23.107980 21475 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:45:23.108045 21475 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.198.2
I20260925 22:45:23.114006 21475 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.198.20:37281
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790376259650642-11032-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790376259650642-11032-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790376259650642-11032-0/krb5kdc/kudu_127.10.198.2.keytab
--principal=kudu/127.10.198.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.198.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790376259650642-11032-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.198.2
--webserver_port=0
--tserver_master_addrs=127.10.198.62:33343
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21475
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.198.2
--log_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790376259650642-11032-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:09:16 UTC on ccycloud.quasar-afmhyw.root.comops.site
build id 4673379
I20260925 22:45:23.115391 21475 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:45:23.116743 21475 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:45:23.128432 21481 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:45:23.128432 21480 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:45:23.128435 21483 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:45:23.128865 21475 server_base.cc:1034] running on GCE node
I20260925 22:45:23.129341 21475 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:45:23.130102 21475 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:45:23.131345 21475 hybrid_clock.cc:648] HybridClock initialized: now 1790376323131307 us; error 62 us; skew 500 ppm
Sep 25 22:45:23 dist-test-slave-5vdr krb5kdc[21234](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376323, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.198.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:45:23.134589 21475 init.cc:377] Logged in from keytab as kudu/127.10.198.2@KRBTEST.COM (short username kudu)
I20260925 22:45:23.135751 21475 webserver.cc:460] Webserver started at http://127.10.198.2:39169/ using document root <none> and password file <none>
I20260925 22:45:23.136462 21475 fs_manager.cc:359] Metadata directory not provided
I20260925 22:45:23.136543 21475 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:45:23.136721 21475 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:45:23.139127 21475 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790376259650642-11032-0/minicluster-data/ts-1/data/instance:
uuid: "11aa0603d7954c32bf623cb954037c1c"
format_stamp: "Formatted at 2026-09-25 22:45:23 on dist-test-slave-5vdr"
server_key: "5df56cdccef78255a094bf73d36c8913"
server_key_iv: "247d4dde687d1ea8675ac33f2e13fd43"
server_key_version: "encryptionkey@0"
I20260925 22:45:23.139787 21475 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790376259650642-11032-0/minicluster-data/ts-1/wal/instance:
uuid: "11aa0603d7954c32bf623cb954037c1c"
format_stamp: "Formatted at 2026-09-25 22:45:23 on dist-test-slave-5vdr"
server_key: "5df56cdccef78255a094bf73d36c8913"
server_key_iv: "247d4dde687d1ea8675ac33f2e13fd43"
server_key_version: "encryptionkey@0"
I20260925 22:45:23.144321 21475 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260925 22:45:23.147287 21490 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:45:23.148967 21475 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 22:45:23.149174 21475 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790376259650642-11032-0/minicluster-data/ts-1/data,/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790376259650642-11032-0/minicluster-data/ts-1/wal
uuid: "11aa0603d7954c32bf623cb954037c1c"
format_stamp: "Formatted at 2026-09-25 22:45:23 on dist-test-slave-5vdr"
server_key: "5df56cdccef78255a094bf73d36c8913"
server_key_iv: "247d4dde687d1ea8675ac33f2e13fd43"
server_key_version: "encryptionkey@0"
I20260925 22:45:23.149330 21475 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790376259650642-11032-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790376259650642-11032-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790376259650642-11032-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:45:23.162253 21475 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:45:23.165875 21475 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:45:23.166118 21475 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:45:23.166810 21475 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 22:45:23.167735 21475 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:45:23.167797 21475 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:45:23.167932 21475 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:45:23.167975 21475 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 22:45:23.179630 21603 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.198.2:46263 every 8 connection(s)
I20260925 22:45:23.179654 21475 rpc_server.cc:307] RPC server started. Bound to: 127.10.198.2:46263
I20260925 22:45:23.180776 21475 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790376259650642-11032-0/minicluster-data/ts-1/data/info.pb
I20260925 22:45:23.188889 11032 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrWXm8X/build/debug/bin/kudu as pid 21475
I20260925 22:45:23.189046 11032 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790376259650642-11032-0/minicluster-data/ts-1/wal/instance
I20260925 22:45:23.189347 11032 external_mini_cluster.cc:1428] Setting key 77df46f6e4dda87f8abe9559f946a339
Sep 25 22:45:23 dist-test-slave-5vdr krb5kdc[21234](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376323, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.198.2@KRBTEST.COM for kudu/127.10.198.62@KRBTEST.COM
I20260925 22:45:23.197854 21318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:45:23.182966 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.198.2:43543 (local address 127.10.198.62:33343)
0925 22:45:23.183245 (+ 279us) server_negotiation.cc:206] Beginning negotiation
0925 22:45:23.183249 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:45:23.184369 (+ 1120us) server_negotiation.cc:407] Connection header received
0925 22:45:23.185873 (+ 1504us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:45:23.185877 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:45:23.185941 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:45:23.186027 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:45:23.187443 (+ 1416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:45:23.188113 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:45:23.189621 (+ 1508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:45:23.189927 (+ 306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:45:23.190084 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:45:23.193288 (+ 3204us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:45:23.193295 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:45:23.193299 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:45:23.193338 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:45:23.195030 (+ 1692us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:45:23.195897 (+ 867us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:45:23.195901 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:45:23.195903 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:45:23.196004 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:45:23.196490 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:45:23.196492 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:45:23.196493 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:45:23.196689 (+ 196us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:45:23.196772 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0925 22:45:23.197592 (+ 820us) server_negotiation.cc:299] Negotiation successful
0925 22:45:23.197720 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":155}
I20260925 22:45:23.199189 21606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:45:23.183407 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.198.62:33343 (local address 127.10.198.2:43543)
0925 22:45:23.184175 (+ 768us) negotiation.cc:107] Waiting for socket to connect
0925 22:45:23.184242 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0925 22:45:23.185599 (+ 1357us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:45:23.186230 (+ 631us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:45:23.186239 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:45:23.186661 (+ 422us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:45:23.187239 (+ 578us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:45:23.187254 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:45:23.188311 (+ 1057us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:45:23.188314 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:45:23.189450 (+ 1136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:45:23.189466 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:45:23.190156 (+ 690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:45:23.190167 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:45:23.190568 (+ 401us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:45:23.191375 (+ 807us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:45:23.191394 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:45:23.193049 (+ 1655us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:45:23.195193 (+ 2144us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:45:23.195205 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:45:23.195229 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:45:23.195725 (+ 496us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:45:23.196117 (+ 392us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:45:23.196123 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:45:23.196126 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:45:23.196377 (+ 251us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:45:23.196807 (+ 430us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:45:23.196815 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:45:23.197253 (+ 438us) client_negotiation.cc:769] Sending connection context
0925 22:45:23.197544 (+ 291us) client_negotiation.cc:240] Negotiation successful
0925 22:45:23.197980 (+ 436us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":378,"thread_start_us":149,"threads_started":1}
WARNING: no policy specified for kudu/127.10.198.3@KRBTEST.COM; defaulting to no policy
I20260925 22:45:23.200659 21604 heartbeater.cc:347] Connected to a master server at 127.10.198.62:33343
I20260925 22:45:23.200954 21604 heartbeater.cc:464] Registering TS with master...
I20260925 22:45:23.201619 21604 heartbeater.cc:511] Master 127.10.198.62:33343 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.198.3@KRBTEST.COM" created.
I20260925 22:45:23.203058 21279 ts_manager.cc:194] Registered new tserver with Master: 11aa0603d7954c32bf623cb954037c1c (127.10.198.2:46263)
I20260925 22:45:23.203698 21279 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.198.2@KRBTEST.COM'} at 127.10.198.2:43543
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.198.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790376259650642-11032-0/krb5kdc/kudu_127.10.198.3.keytab.
Entry for principal kudu/127.10.198.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790376259650642-11032-0/krb5kdc/kudu_127.10.198.3.keytab.
I20260925 22:45:23.219295 11032 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrWXm8X/build/debug/bin/kudu
/tmp/dist-test-taskrWXm8X/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790376259650642-11032-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790376259650642-11032-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790376259650642-11032-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790376259650642-11032-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.198.3:0
--local_ip_for_outbound_sockets=127.10.198.3
--webserver_interface=127.10.198.3
--webserver_port=0
--tserver_master_addrs=127.10.198.62:33343
--keytab_file=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790376259650642-11032-0/krb5kdc/kudu_127.10.198.3.keytab
--principal=kudu/127.10.198.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.198.20:37281
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790376259650642-11032-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790376259650642-11032-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790376259650642-11032-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:45:23.342350 21609 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:45:23.342724 21609 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:45:23.342804 21609 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:45:23.347378 21609 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:45:23.347497 21609 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:45:23.347601 21609 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.198.3
I20260925 22:45:23.353747 21609 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.198.20:37281
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790376259650642-11032-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790376259650642-11032-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790376259650642-11032-0/krb5kdc/kudu_127.10.198.3.keytab
--principal=kudu/127.10.198.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.198.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790376259650642-11032-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.198.3
--webserver_port=0
--tserver_master_addrs=127.10.198.62:33343
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21609
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.198.3
--log_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790376259650642-11032-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:09:16 UTC on ccycloud.quasar-afmhyw.root.comops.site
build id 4673379
I20260925 22:45:23.355059 21609 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:45:23.356393 21609 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:45:23.367317 21615 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:45:23.367416 21609 server_base.cc:1034] running on GCE node
W20260925 22:45:23.367218 21617 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:45:23.367240 21614 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:45:23.367997 21609 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:45:23.368769 21609 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:45:23.369988 21609 hybrid_clock.cc:648] HybridClock initialized: now 1790376323369973 us; error 43 us; skew 500 ppm
Sep 25 22:45:23 dist-test-slave-5vdr krb5kdc[21234](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376323, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.198.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:45:23.373231 21609 init.cc:377] Logged in from keytab as kudu/127.10.198.3@KRBTEST.COM (short username kudu)
I20260925 22:45:23.374240 21609 webserver.cc:460] Webserver started at http://127.10.198.3:35999/ using document root <none> and password file <none>
I20260925 22:45:23.374851 21609 fs_manager.cc:359] Metadata directory not provided
I20260925 22:45:23.374933 21609 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:45:23.375118 21609 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:45:23.377799 21609 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790376259650642-11032-0/minicluster-data/ts-2/data/instance:
uuid: "85e9146c658b4ab480aa7b5fffc53357"
format_stamp: "Formatted at 2026-09-25 22:45:23 on dist-test-slave-5vdr"
server_key: "281eb33d479b0cc8c8ebb3f6c44e6b75"
server_key_iv: "2355b4812626aefc45b6389aa7179ecf"
server_key_version: "encryptionkey@0"
I20260925 22:45:23.378541 21609 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790376259650642-11032-0/minicluster-data/ts-2/wal/instance:
uuid: "85e9146c658b4ab480aa7b5fffc53357"
format_stamp: "Formatted at 2026-09-25 22:45:23 on dist-test-slave-5vdr"
server_key: "281eb33d479b0cc8c8ebb3f6c44e6b75"
server_key_iv: "2355b4812626aefc45b6389aa7179ecf"
server_key_version: "encryptionkey@0"
I20260925 22:45:23.382931 21609 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260925 22:45:23.385866 21624 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:45:23.387449 21609 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260925 22:45:23.387619 21609 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790376259650642-11032-0/minicluster-data/ts-2/data,/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790376259650642-11032-0/minicluster-data/ts-2/wal
uuid: "85e9146c658b4ab480aa7b5fffc53357"
format_stamp: "Formatted at 2026-09-25 22:45:23 on dist-test-slave-5vdr"
server_key: "281eb33d479b0cc8c8ebb3f6c44e6b75"
server_key_iv: "2355b4812626aefc45b6389aa7179ecf"
server_key_version: "encryptionkey@0"
I20260925 22:45:23.387760 21609 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790376259650642-11032-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790376259650642-11032-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790376259650642-11032-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:45:23.428582 21609 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:45:23.432466 21609 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:45:23.432713 21609 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:45:23.433400 21609 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 22:45:23.434360 21609 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:45:23.434410 21609 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:45:23.434489 21609 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:45:23.434535 21609 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 22:45:23.444800 21609 rpc_server.cc:307] RPC server started. Bound to: 127.10.198.3:37315
I20260925 22:45:23.444839 21737 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.198.3:37315 every 8 connection(s)
I20260925 22:45:23.445883 21609 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790376259650642-11032-0/minicluster-data/ts-2/data/info.pb
I20260925 22:45:23.448114 11032 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrWXm8X/build/debug/bin/kudu as pid 21609
I20260925 22:45:23.448261 11032 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790376259650642-11032-0/minicluster-data/ts-2/wal/instance
I20260925 22:45:23.448599 11032 external_mini_cluster.cc:1428] Setting key 023499176db126e2e2c199dcee64415f
Sep 25 22:45:23 dist-test-slave-5vdr krb5kdc[21234](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376323, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.198.3@KRBTEST.COM for kudu/127.10.198.62@KRBTEST.COM
I20260925 22:45:23.463127 21318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:45:23.448236 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.198.3:41821 (local address 127.10.198.62:33343)
0925 22:45:23.448399 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0925 22:45:23.448403 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:45:23.449756 (+ 1353us) server_negotiation.cc:407] Connection header received
0925 22:45:23.451682 (+ 1926us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:45:23.451685 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:45:23.451737 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:45:23.451815 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:45:23.453292 (+ 1477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:45:23.453879 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:45:23.455241 (+ 1362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:45:23.455500 (+ 259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:45:23.455596 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:45:23.458744 (+ 3148us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:45:23.458748 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:45:23.458750 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:45:23.458789 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:45:23.460562 (+ 1773us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:45:23.461206 (+ 644us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:45:23.461208 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:45:23.461210 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:45:23.461296 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:45:23.461692 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:45:23.461695 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:45:23.461697 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:45:23.461959 (+ 262us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:45:23.462064 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0925 22:45:23.462831 (+ 767us) server_negotiation.cc:299] Negotiation successful
0925 22:45:23.462983 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260925 22:45:23.464234 21740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:45:23.448866 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.198.62:33343 (local address 127.10.198.3:41821)
0925 22:45:23.449555 (+ 689us) negotiation.cc:107] Waiting for socket to connect
0925 22:45:23.449643 (+ 88us) client_negotiation.cc:174] Beginning negotiation
0925 22:45:23.451452 (+ 1809us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:45:23.452048 (+ 596us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:45:23.452056 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:45:23.452475 (+ 419us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:45:23.453025 (+ 550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:45:23.453042 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:45:23.454031 (+ 989us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:45:23.454034 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:45:23.455084 (+ 1050us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:45:23.455097 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:45:23.455667 (+ 570us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:45:23.455670 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:45:23.455940 (+ 270us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:45:23.456662 (+ 722us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:45:23.456683 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:45:23.458494 (+ 1811us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:45:23.460729 (+ 2235us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:45:23.460736 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:45:23.460752 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:45:23.461094 (+ 342us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:45:23.461399 (+ 305us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:45:23.461402 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:45:23.461404 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:45:23.461568 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:45:23.462078 (+ 510us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:45:23.462085 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:45:23.462508 (+ 423us) client_negotiation.cc:769] Sending connection context
0925 22:45:23.462740 (+ 232us) client_negotiation.cc:240] Negotiation successful
0925 22:45:23.463169 (+ 429us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":396,"thread_start_us":204,"threads_started":1}
I20260925 22:45:23.465723 21738 heartbeater.cc:347] Connected to a master server at 127.10.198.62:33343
I20260925 22:45:23.466090 21738 heartbeater.cc:464] Registering TS with master...
I20260925 22:45:23.466729 21738 heartbeater.cc:511] Master 127.10.198.62:33343 requested a full tablet report, sending...
I20260925 22:45:23.468143 21279 ts_manager.cc:194] Registered new tserver with Master: 85e9146c658b4ab480aa7b5fffc53357 (127.10.198.3:37315)
I20260925 22:45:23.468828 21279 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.198.3@KRBTEST.COM'} at 127.10.198.3:41821
I20260925 22:45:23.476440 11032 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 22:45:23.487460 21318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:45:23.478974 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54958 (local address 127.10.198.62:33343)
0925 22:45:23.479152 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0925 22:45:23.479156 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:45:23.479527 (+ 371us) server_negotiation.cc:407] Connection header received
0925 22:45:23.479728 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:45:23.479738 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:45:23.479789 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:45:23.479919 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:45:23.480716 (+ 797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:45:23.481256 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:45:23.482141 (+ 885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:45:23.482368 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:45:23.482492 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:45:23.484104 (+ 1612us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:45:23.484107 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:45:23.484109 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:45:23.484140 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:45:23.485642 (+ 1502us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:45:23.486118 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:45:23.486122 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:45:23.486123 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:45:23.486212 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:45:23.486514 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:45:23.486520 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:45:23.486522 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:45:23.486766 (+ 244us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:45:23.486912 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0925 22:45:23.487156 (+ 244us) server_negotiation.cc:299] Negotiation successful
0925 22:45:23.487285 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260925 22:45:23.490968 21279 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54958:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 22:45:23.494999 21279 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 22:45:23.511512 21749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:45:23.504998 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.198.1:38947 (local address 127.0.0.1:34052)
0925 22:45:23.505598 (+ 600us) negotiation.cc:107] Waiting for socket to connect
0925 22:45:23.505665 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0925 22:45:23.505812 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:45:23.506572 (+ 760us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:45:23.506576 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:45:23.506617 (+ 41us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:45:23.506769 (+ 152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:45:23.506777 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:45:23.508368 (+ 1591us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:45:23.508371 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:45:23.509902 (+ 1531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:45:23.509910 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:45:23.510936 (+ 1026us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:45:23.510940 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:45:23.511071 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:45:23.511122 (+ 51us) client_negotiation.cc:769] Sending connection context
0925 22:45:23.511248 (+ 126us) client_negotiation.cc:240] Negotiation successful
0925 22:45:23.511367 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":444,"mutex_wait_us":22,"spinlock_wait_cycles":24704,"thread_start_us":107,"threads_started":1}
I20260925 22:45:23.511629 21752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:45:23.505144 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34052 (local address 127.10.198.1:38947)
0925 22:45:23.506096 (+ 952us) server_negotiation.cc:206] Beginning negotiation
0925 22:45:23.506103 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 22:45:23.506126 (+ 23us) server_negotiation.cc:407] Connection header received
0925 22:45:23.506202 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:45:23.506209 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:45:23.506403 (+ 194us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:45:23.506917 (+ 514us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:45:23.507059 (+ 142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:45:23.508241 (+ 1182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:45:23.510039 (+ 1798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:45:23.510777 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:45:23.511021 (+ 244us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:45:23.511190 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0925 22:45:23.511280 (+ 90us) server_negotiation.cc:299] Negotiation successful
0925 22:45:23.511409 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":850,"thread_start_us":75,"threads_started":1}
I20260925 22:45:23.512461 21751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:45:23.504766 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.198.2:46263 (local address 127.0.0.1:33830)
0925 22:45:23.505598 (+ 832us) negotiation.cc:107] Waiting for socket to connect
0925 22:45:23.505665 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0925 22:45:23.505812 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:45:23.506566 (+ 754us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:45:23.506572 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:45:23.506602 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:45:23.506759 (+ 157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:45:23.506765 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:45:23.508142 (+ 1377us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:45:23.508146 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:45:23.509659 (+ 1513us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:45:23.509671 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:45:23.510886 (+ 1215us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:45:23.510890 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:45:23.512094 (+ 1204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:45:23.512121 (+ 27us) client_negotiation.cc:769] Sending connection context
0925 22:45:23.512204 (+ 83us) client_negotiation.cc:240] Negotiation successful
0925 22:45:23.512294 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":638,"mutex_wait_us":6,"spinlock_wait_cycles":15232,"thread_start_us":93,"threads_started":1}
I20260925 22:45:23.512956 21750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:45:23.504858 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33830 (local address 127.10.198.2:46263)
0925 22:45:23.505794 (+ 936us) server_negotiation.cc:206] Beginning negotiation
0925 22:45:23.505809 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0925 22:45:23.505840 (+ 31us) server_negotiation.cc:407] Connection header received
0925 22:45:23.505962 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:45:23.505967 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:45:23.506233 (+ 266us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:45:23.506479 (+ 246us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:45:23.506904 (+ 425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:45:23.507988 (+ 1084us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:45:23.509816 (+ 1828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:45:23.510742 (+ 926us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:45:23.510878 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:45:23.511057 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0925 22:45:23.512657 (+ 1600us) server_negotiation.cc:299] Negotiation successful
0925 22:45:23.512780 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":821,"thread_start_us":108,"threads_started":1}
I20260925 22:45:23.515137 21753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:45:23.505580 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.198.3:37315 (local address 127.0.0.1:36422)
0925 22:45:23.506559 (+ 979us) negotiation.cc:107] Waiting for socket to connect
0925 22:45:23.506573 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 22:45:23.506663 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:45:23.507260 (+ 597us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:45:23.507263 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:45:23.507286 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:45:23.507380 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:45:23.507390 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:45:23.508980 (+ 1590us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:45:23.508983 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:45:23.510395 (+ 1412us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:45:23.510407 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:45:23.514518 (+ 4111us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:45:23.514523 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:45:23.514648 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:45:23.514678 (+ 30us) client_negotiation.cc:769] Sending connection context
0925 22:45:23.514777 (+ 99us) client_negotiation.cc:240] Negotiation successful
0925 22:45:23.514883 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":899,"spinlock_wait_cycles":1408,"thread_start_us":119,"threads_started":1}
I20260925 22:45:23.516047 21404 tablet_service.cc:1467] Processing CreateTablet for tablet 0acde504c69947e991398abfd6fea6d2 (DEFAULT_TABLE table=test-table [id=1b0d2fb1d4754f59b3dc45e834573c60]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 22:45:23.517580 21404 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0acde504c69947e991398abfd6fea6d2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:45:23.516273 21754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:45:23.506018 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36422 (local address 127.10.198.3:37315)
0925 22:45:23.506441 (+ 423us) server_negotiation.cc:206] Beginning negotiation
0925 22:45:23.506452 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 22:45:23.506758 (+ 306us) server_negotiation.cc:407] Connection header received
0925 22:45:23.506845 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:45:23.506853 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:45:23.507071 (+ 218us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:45:23.507235 (+ 164us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:45:23.507523 (+ 288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:45:23.508599 (+ 1076us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:45:23.510559 (+ 1960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:45:23.513989 (+ 3430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:45:23.514141 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:45:23.514307 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0925 22:45:23.515370 (+ 1063us) server_negotiation.cc:299] Negotiation successful
0925 22:45:23.515611 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":334,"thread_start_us":139,"threads_started":1}
I20260925 22:45:23.519645 21538 tablet_service.cc:1467] Processing CreateTablet for tablet 0acde504c69947e991398abfd6fea6d2 (DEFAULT_TABLE table=test-table [id=1b0d2fb1d4754f59b3dc45e834573c60]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 22:45:23.521176 21538 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0acde504c69947e991398abfd6fea6d2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:45:23.523556 21672 tablet_service.cc:1467] Processing CreateTablet for tablet 0acde504c69947e991398abfd6fea6d2 (DEFAULT_TABLE table=test-table [id=1b0d2fb1d4754f59b3dc45e834573c60]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 22:45:23.524951 21672 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0acde504c69947e991398abfd6fea6d2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:45:23.526496 21755 tablet_bootstrap.cc:492] T 0acde504c69947e991398abfd6fea6d2 P 54d82bc61dea4af89266201d8a1deb47: Bootstrap starting.
I20260925 22:45:23.529856 21755 tablet_bootstrap.cc:654] T 0acde504c69947e991398abfd6fea6d2 P 54d82bc61dea4af89266201d8a1deb47: Neither blocks nor log segments found. Creating new log.
I20260925 22:45:23.529891 21756 tablet_bootstrap.cc:492] T 0acde504c69947e991398abfd6fea6d2 P 11aa0603d7954c32bf623cb954037c1c: Bootstrap starting.
I20260925 22:45:23.530866 21755 log.cc:824] T 0acde504c69947e991398abfd6fea6d2 P 54d82bc61dea4af89266201d8a1deb47: Log is configured to *not* fsync() on all Append() calls
I20260925 22:45:23.533188 21756 tablet_bootstrap.cc:654] T 0acde504c69947e991398abfd6fea6d2 P 11aa0603d7954c32bf623cb954037c1c: Neither blocks nor log segments found. Creating new log.
I20260925 22:45:23.533897 21758 tablet_bootstrap.cc:492] T 0acde504c69947e991398abfd6fea6d2 P 85e9146c658b4ab480aa7b5fffc53357: Bootstrap starting.
I20260925 22:45:23.534292 21755 tablet_bootstrap.cc:492] T 0acde504c69947e991398abfd6fea6d2 P 54d82bc61dea4af89266201d8a1deb47: No bootstrap required, opened a new log
I20260925 22:45:23.534539 21755 ts_tablet_manager.cc:1397] T 0acde504c69947e991398abfd6fea6d2 P 54d82bc61dea4af89266201d8a1deb47: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260925 22:45:23.534890 21756 log.cc:824] T 0acde504c69947e991398abfd6fea6d2 P 11aa0603d7954c32bf623cb954037c1c: Log is configured to *not* fsync() on all Append() calls
I20260925 22:45:23.536751 21758 tablet_bootstrap.cc:654] T 0acde504c69947e991398abfd6fea6d2 P 85e9146c658b4ab480aa7b5fffc53357: Neither blocks nor log segments found. Creating new log.
I20260925 22:45:23.537753 21756 tablet_bootstrap.cc:492] T 0acde504c69947e991398abfd6fea6d2 P 11aa0603d7954c32bf623cb954037c1c: No bootstrap required, opened a new log
I20260925 22:45:23.537973 21758 log.cc:824] T 0acde504c69947e991398abfd6fea6d2 P 85e9146c658b4ab480aa7b5fffc53357: Log is configured to *not* fsync() on all Append() calls
I20260925 22:45:23.538043 21756 ts_tablet_manager.cc:1397] T 0acde504c69947e991398abfd6fea6d2 P 11aa0603d7954c32bf623cb954037c1c: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20260925 22:45:23.539355 21755 raft_consensus.cc:348] T 0acde504c69947e991398abfd6fea6d2 P 54d82bc61dea4af89266201d8a1deb47 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11aa0603d7954c32bf623cb954037c1c" member_type: VOTER last_known_addr { host: "127.10.198.2" port: 46263 } } peers { permanent_uuid: "85e9146c658b4ab480aa7b5fffc53357" member_type: VOTER last_known_addr { host: "127.10.198.3" port: 37315 } } peers { permanent_uuid: "54d82bc61dea4af89266201d8a1deb47" member_type: VOTER last_known_addr { host: "127.10.198.1" port: 38947 } }
I20260925 22:45:23.539729 21755 raft_consensus.cc:374] T 0acde504c69947e991398abfd6fea6d2 P 54d82bc61dea4af89266201d8a1deb47 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:45:23.539788 21755 raft_consensus.cc:729] T 0acde504c69947e991398abfd6fea6d2 P 54d82bc61dea4af89266201d8a1deb47 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 54d82bc61dea4af89266201d8a1deb47, State: Initialized, Role: FOLLOWER
I20260925 22:45:23.540661 21755 consensus_queue.cc:260] T 0acde504c69947e991398abfd6fea6d2 P 54d82bc61dea4af89266201d8a1deb47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11aa0603d7954c32bf623cb954037c1c" member_type: VOTER last_known_addr { host: "127.10.198.2" port: 46263 } } peers { permanent_uuid: "85e9146c658b4ab480aa7b5fffc53357" member_type: VOTER last_known_addr { host: "127.10.198.3" port: 37315 } } peers { permanent_uuid: "54d82bc61dea4af89266201d8a1deb47" member_type: VOTER last_known_addr { host: "127.10.198.1" port: 38947 } }
I20260925 22:45:23.541249 21758 tablet_bootstrap.cc:492] T 0acde504c69947e991398abfd6fea6d2 P 85e9146c658b4ab480aa7b5fffc53357: No bootstrap required, opened a new log
I20260925 22:45:23.541488 21758 ts_tablet_manager.cc:1397] T 0acde504c69947e991398abfd6fea6d2 P 85e9146c658b4ab480aa7b5fffc53357: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260925 22:45:23.541909 21755 ts_tablet_manager.cc:1428] T 0acde504c69947e991398abfd6fea6d2 P 54d82bc61dea4af89266201d8a1deb47: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260925 22:45:23.542342 21470 heartbeater.cc:503] Master 127.10.198.62:33343 was elected leader, sending a full tablet report...
I20260925 22:45:23.542568 21756 raft_consensus.cc:348] T 0acde504c69947e991398abfd6fea6d2 P 11aa0603d7954c32bf623cb954037c1c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11aa0603d7954c32bf623cb954037c1c" member_type: VOTER last_known_addr { host: "127.10.198.2" port: 46263 } } peers { permanent_uuid: "85e9146c658b4ab480aa7b5fffc53357" member_type: VOTER last_known_addr { host: "127.10.198.3" port: 37315 } } peers { permanent_uuid: "54d82bc61dea4af89266201d8a1deb47" member_type: VOTER last_known_addr { host: "127.10.198.1" port: 38947 } }
I20260925 22:45:23.542799 21756 raft_consensus.cc:374] T 0acde504c69947e991398abfd6fea6d2 P 11aa0603d7954c32bf623cb954037c1c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:45:23.542838 21756 raft_consensus.cc:729] T 0acde504c69947e991398abfd6fea6d2 P 11aa0603d7954c32bf623cb954037c1c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 11aa0603d7954c32bf623cb954037c1c, State: Initialized, Role: FOLLOWER
I20260925 22:45:23.543393 21756 consensus_queue.cc:260] T 0acde504c69947e991398abfd6fea6d2 P 11aa0603d7954c32bf623cb954037c1c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11aa0603d7954c32bf623cb954037c1c" member_type: VOTER last_known_addr { host: "127.10.198.2" port: 46263 } } peers { permanent_uuid: "85e9146c658b4ab480aa7b5fffc53357" member_type: VOTER last_known_addr { host: "127.10.198.3" port: 37315 } } peers { permanent_uuid: "54d82bc61dea4af89266201d8a1deb47" member_type: VOTER last_known_addr { host: "127.10.198.1" port: 38947 } }
I20260925 22:45:23.544253 21604 heartbeater.cc:503] Master 127.10.198.62:33343 was elected leader, sending a full tablet report...
I20260925 22:45:23.544641 21756 ts_tablet_manager.cc:1428] T 0acde504c69947e991398abfd6fea6d2 P 11aa0603d7954c32bf623cb954037c1c: Time spent starting tablet: real 0.007s user 0.005s sys 0.000s
I20260925 22:45:23.545492 21758 raft_consensus.cc:348] T 0acde504c69947e991398abfd6fea6d2 P 85e9146c658b4ab480aa7b5fffc53357 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11aa0603d7954c32bf623cb954037c1c" member_type: VOTER last_known_addr { host: "127.10.198.2" port: 46263 } } peers { permanent_uuid: "85e9146c658b4ab480aa7b5fffc53357" member_type: VOTER last_known_addr { host: "127.10.198.3" port: 37315 } } peers { permanent_uuid: "54d82bc61dea4af89266201d8a1deb47" member_type: VOTER last_known_addr { host: "127.10.198.1" port: 38947 } }
I20260925 22:45:23.545814 21758 raft_consensus.cc:374] T 0acde504c69947e991398abfd6fea6d2 P 85e9146c658b4ab480aa7b5fffc53357 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:45:23.545881 21758 raft_consensus.cc:729] T 0acde504c69947e991398abfd6fea6d2 P 85e9146c658b4ab480aa7b5fffc53357 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85e9146c658b4ab480aa7b5fffc53357, State: Initialized, Role: FOLLOWER
I20260925 22:45:23.546434 21758 consensus_queue.cc:260] T 0acde504c69947e991398abfd6fea6d2 P 85e9146c658b4ab480aa7b5fffc53357 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11aa0603d7954c32bf623cb954037c1c" member_type: VOTER last_known_addr { host: "127.10.198.2" port: 46263 } } peers { permanent_uuid: "85e9146c658b4ab480aa7b5fffc53357" member_type: VOTER last_known_addr { host: "127.10.198.3" port: 37315 } } peers { permanent_uuid: "54d82bc61dea4af89266201d8a1deb47" member_type: VOTER last_known_addr { host: "127.10.198.1" port: 38947 } }
I20260925 22:45:23.547210 21738 heartbeater.cc:503] Master 127.10.198.62:33343 was elected leader, sending a full tablet report...
I20260925 22:45:23.547660 21758 ts_tablet_manager.cc:1428] T 0acde504c69947e991398abfd6fea6d2 P 85e9146c658b4ab480aa7b5fffc53357: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
W20260925 22:45:23.682116 21605 tablet.cc:2369] T 0acde504c69947e991398abfd6fea6d2 P 11aa0603d7954c32bf623cb954037c1c: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 22:45:23.696300 21471 tablet.cc:2369] T 0acde504c69947e991398abfd6fea6d2 P 54d82bc61dea4af89266201d8a1deb47: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 22:45:23.697057 21739 tablet.cc:2369] T 0acde504c69947e991398abfd6fea6d2 P 85e9146c658b4ab480aa7b5fffc53357: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 22:45:23.853080 21762 raft_consensus.cc:482] T 0acde504c69947e991398abfd6fea6d2 P 11aa0603d7954c32bf623cb954037c1c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 22:45:23.853299 21762 raft_consensus.cc:504] T 0acde504c69947e991398abfd6fea6d2 P 11aa0603d7954c32bf623cb954037c1c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11aa0603d7954c32bf623cb954037c1c" member_type: VOTER last_known_addr { host: "127.10.198.2" port: 46263 } } peers { permanent_uuid: "85e9146c658b4ab480aa7b5fffc53357" member_type: VOTER last_known_addr { host: "127.10.198.3" port: 37315 } } peers { permanent_uuid: "54d82bc61dea4af89266201d8a1deb47" member_type: VOTER last_known_addr { host: "127.10.198.1" port: 38947 } }
I20260925 22:45:23.854538 21762 leader_election.cc:288] T 0acde504c69947e991398abfd6fea6d2 P 11aa0603d7954c32bf623cb954037c1c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 85e9146c658b4ab480aa7b5fffc53357 (127.10.198.3:37315), 54d82bc61dea4af89266201d8a1deb47 (127.10.198.1:38947)
I20260925 22:45:23.859051 21752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:45:23.854927 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.198.2:56749 (local address 127.10.198.1:38947)
0925 22:45:23.855059 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0925 22:45:23.855064 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:45:23.855422 (+ 358us) server_negotiation.cc:407] Connection header received
0925 22:45:23.855590 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:45:23.855593 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:45:23.855651 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:45:23.855744 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:45:23.856065 (+ 321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:45:23.856583 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:45:23.857939 (+ 1356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:45:23.858386 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:45:23.858536 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:45:23.858600 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0925 22:45:23.858809 (+ 209us) server_negotiation.cc:299] Negotiation successful
0925 22:45:23.858895 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260925 22:45:23.859052 21754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:45:23.854944 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.198.2:34467 (local address 127.10.198.3:37315)
0925 22:45:23.855085 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0925 22:45:23.855091 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 22:45:23.855329 (+ 238us) server_negotiation.cc:407] Connection header received
0925 22:45:23.855508 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:45:23.855511 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:45:23.855565 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:45:23.855664 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:45:23.856053 (+ 389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:45:23.856584 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:45:23.857947 (+ 1363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:45:23.858374 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:45:23.858518 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:45:23.858577 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0925 22:45:23.858811 (+ 234us) server_negotiation.cc:299] Negotiation successful
0925 22:45:23.858892 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260925 22:45:23.859113 21764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:45:23.854810 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.198.3:37315 (local address 127.10.198.2:34467)
0925 22:45:23.855198 (+ 388us) negotiation.cc:107] Waiting for socket to connect
0925 22:45:23.855211 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 22:45:23.855384 (+ 173us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:45:23.855694 (+ 310us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:45:23.855697 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:45:23.855714 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:45:23.855928 (+ 214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:45:23.855936 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:45:23.856718 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:45:23.856723 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:45:23.857809 (+ 1086us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:45:23.857819 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:45:23.858518 (+ 699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:45:23.858522 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:45:23.858647 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:45:23.858700 (+ 53us) client_negotiation.cc:769] Sending connection context
0925 22:45:23.858794 (+ 94us) client_negotiation.cc:240] Negotiation successful
0925 22:45:23.858892 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":320,"mutex_wait_us":79,"thread_start_us":93,"threads_started":1}
I20260925 22:45:23.859097 21765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:45:23.854810 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.198.1:38947 (local address 127.10.198.2:56749)
0925 22:45:23.855282 (+ 472us) negotiation.cc:107] Waiting for socket to connect
0925 22:45:23.855295 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 22:45:23.855465 (+ 170us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:45:23.855764 (+ 299us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:45:23.855767 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:45:23.855783 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:45:23.855937 (+ 154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:45:23.855943 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:45:23.856713 (+ 770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:45:23.856718 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:45:23.857798 (+ 1080us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:45:23.857809 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:45:23.858520 (+ 711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:45:23.858524 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:45:23.858680 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:45:23.858699 (+ 19us) client_negotiation.cc:769] Sending connection context
0925 22:45:23.858793 (+ 94us) client_negotiation.cc:240] Negotiation successful
0925 22:45:23.858894 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":414,"mutex_wait_us":82,"thread_start_us":82,"threads_started":1}
I20260925 22:45:23.859948 21692 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0acde504c69947e991398abfd6fea6d2" candidate_uuid: "11aa0603d7954c32bf623cb954037c1c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85e9146c658b4ab480aa7b5fffc53357" is_pre_election: true
I20260925 22:45:23.860050 21424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0acde504c69947e991398abfd6fea6d2" candidate_uuid: "11aa0603d7954c32bf623cb954037c1c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "54d82bc61dea4af89266201d8a1deb47" is_pre_election: true
I20260925 22:45:23.860325 21692 raft_consensus.cc:2454] T 0acde504c69947e991398abfd6fea6d2 P 85e9146c658b4ab480aa7b5fffc53357 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 11aa0603d7954c32bf623cb954037c1c in term 0.
I20260925 22:45:23.860396 21424 raft_consensus.cc:2454] T 0acde504c69947e991398abfd6fea6d2 P 54d82bc61dea4af89266201d8a1deb47 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 11aa0603d7954c32bf623cb954037c1c in term 0.
I20260925 22:45:23.860957 21494 leader_election.cc:302] T 0acde504c69947e991398abfd6fea6d2 P 11aa0603d7954c32bf623cb954037c1c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 11aa0603d7954c32bf623cb954037c1c, 54d82bc61dea4af89266201d8a1deb47, 85e9146c658b4ab480aa7b5fffc53357; no voters:
I20260925 22:45:23.861320 21762 raft_consensus.cc:2781] T 0acde504c69947e991398abfd6fea6d2 P 11aa0603d7954c32bf623cb954037c1c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 22:45:23.861410 21762 raft_consensus.cc:482] T 0acde504c69947e991398abfd6fea6d2 P 11aa0603d7954c32bf623cb954037c1c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 22:45:23.861536 21762 raft_consensus.cc:3037] T 0acde504c69947e991398abfd6fea6d2 P 11aa0603d7954c32bf623cb954037c1c [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:45:23.862640 21762 raft_consensus.cc:504] T 0acde504c69947e991398abfd6fea6d2 P 11aa0603d7954c32bf623cb954037c1c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11aa0603d7954c32bf623cb954037c1c" member_type: VOTER last_known_addr { host: "127.10.198.2" port: 46263 } } peers { permanent_uuid: "85e9146c658b4ab480aa7b5fffc53357" member_type: VOTER last_known_addr { host: "127.10.198.3" port: 37315 } } peers { permanent_uuid: "54d82bc61dea4af89266201d8a1deb47" member_type: VOTER last_known_addr { host: "127.10.198.1" port: 38947 } }
I20260925 22:45:23.863118 21762 leader_election.cc:288] T 0acde504c69947e991398abfd6fea6d2 P 11aa0603d7954c32bf623cb954037c1c [CANDIDATE]: Term 1 election: Requested vote from peers 85e9146c658b4ab480aa7b5fffc53357 (127.10.198.3:37315), 54d82bc61dea4af89266201d8a1deb47 (127.10.198.1:38947)
I20260925 22:45:23.864117 21692 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0acde504c69947e991398abfd6fea6d2" candidate_uuid: "11aa0603d7954c32bf623cb954037c1c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85e9146c658b4ab480aa7b5fffc53357"
I20260925 22:45:23.864255 21424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0acde504c69947e991398abfd6fea6d2" candidate_uuid: "11aa0603d7954c32bf623cb954037c1c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "54d82bc61dea4af89266201d8a1deb47"
I20260925 22:45:23.864296 21692 raft_consensus.cc:3037] T 0acde504c69947e991398abfd6fea6d2 P 85e9146c658b4ab480aa7b5fffc53357 [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:45:23.864416 21424 raft_consensus.cc:3037] T 0acde504c69947e991398abfd6fea6d2 P 54d82bc61dea4af89266201d8a1deb47 [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:45:23.865593 21424 raft_consensus.cc:2454] T 0acde504c69947e991398abfd6fea6d2 P 54d82bc61dea4af89266201d8a1deb47 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 11aa0603d7954c32bf623cb954037c1c in term 1.
I20260925 22:45:23.866082 21493 leader_election.cc:302] T 0acde504c69947e991398abfd6fea6d2 P 11aa0603d7954c32bf623cb954037c1c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 11aa0603d7954c32bf623cb954037c1c, 54d82bc61dea4af89266201d8a1deb47; no voters:
I20260925 22:45:23.866156 21692 raft_consensus.cc:2454] T 0acde504c69947e991398abfd6fea6d2 P 85e9146c658b4ab480aa7b5fffc53357 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 11aa0603d7954c32bf623cb954037c1c in term 1.
I20260925 22:45:23.866343 21762 raft_consensus.cc:2781] T 0acde504c69947e991398abfd6fea6d2 P 11aa0603d7954c32bf623cb954037c1c [term 1 FOLLOWER]: Leader election won for term 1
I20260925 22:45:23.866843 21762 raft_consensus.cc:686] T 0acde504c69947e991398abfd6fea6d2 P 11aa0603d7954c32bf623cb954037c1c [term 1 LEADER]: Becoming Leader. State: Replica: 11aa0603d7954c32bf623cb954037c1c, State: Running, Role: LEADER
I20260925 22:45:23.867403 21762 consensus_queue.cc:237] T 0acde504c69947e991398abfd6fea6d2 P 11aa0603d7954c32bf623cb954037c1c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11aa0603d7954c32bf623cb954037c1c" member_type: VOTER last_known_addr { host: "127.10.198.2" port: 46263 } } peers { permanent_uuid: "85e9146c658b4ab480aa7b5fffc53357" member_type: VOTER last_known_addr { host: "127.10.198.3" port: 37315 } } peers { permanent_uuid: "54d82bc61dea4af89266201d8a1deb47" member_type: VOTER last_known_addr { host: "127.10.198.1" port: 38947 } }
I20260925 22:45:23.874965 21279 catalog_manager.cc:5495] T 0acde504c69947e991398abfd6fea6d2 P 11aa0603d7954c32bf623cb954037c1c reported cstate change: term changed from 0 to 1, leader changed from <none> to 11aa0603d7954c32bf623cb954037c1c (127.10.198.2). New cstate: current_term: 1 leader_uuid: "11aa0603d7954c32bf623cb954037c1c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11aa0603d7954c32bf623cb954037c1c" member_type: VOTER last_known_addr { host: "127.10.198.2" port: 46263 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "85e9146c658b4ab480aa7b5fffc53357" member_type: VOTER last_known_addr { host: "127.10.198.3" port: 37315 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "54d82bc61dea4af89266201d8a1deb47" member_type: VOTER last_known_addr { host: "127.10.198.1" port: 38947 } health_report { overall_health: UNKNOWN } } }
I20260925 22:45:23.972671 21750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:45:23.968831 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33834 (local address 127.10.198.2:46263)
0925 22:45:23.968966 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0925 22:45:23.968970 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:45:23.968986 (+ 16us) server_negotiation.cc:407] Connection header received
0925 22:45:23.969134 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:45:23.969137 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:45:23.969208 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:45:23.969334 (+ 126us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 22:45:23.969598 (+ 264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:45:23.970111 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:45:23.971032 (+ 921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:45:23.971267 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:45:23.971389 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:45:23.971730 (+ 341us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 22:45:23.972224 (+ 494us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 22:45:23.972368 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0925 22:45:23.972439 (+ 71us) server_negotiation.cc:299] Negotiation successful
0925 22:45:23.972508 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260925 22:45:23.981122 21692 raft_consensus.cc:1261] T 0acde504c69947e991398abfd6fea6d2 P 85e9146c658b4ab480aa7b5fffc53357 [term 1 FOLLOWER]: Refusing update from remote peer 11aa0603d7954c32bf623cb954037c1c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 22:45:23.981122 21424 raft_consensus.cc:1261] T 0acde504c69947e991398abfd6fea6d2 P 54d82bc61dea4af89266201d8a1deb47 [term 1 FOLLOWER]: Refusing update from remote peer 11aa0603d7954c32bf623cb954037c1c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 22:45:23.982226 21762 consensus_queue.cc:1035] T 0acde504c69947e991398abfd6fea6d2 P 11aa0603d7954c32bf623cb954037c1c [LEADER]: Connected to new peer: Peer: permanent_uuid: "85e9146c658b4ab480aa7b5fffc53357" member_type: VOTER last_known_addr { host: "127.10.198.3" port: 37315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 22:45:23.982496 21766 consensus_queue.cc:1035] T 0acde504c69947e991398abfd6fea6d2 P 11aa0603d7954c32bf623cb954037c1c [LEADER]: Connected to new peer: Peer: permanent_uuid: "54d82bc61dea4af89266201d8a1deb47" member_type: VOTER last_known_addr { host: "127.10.198.1" port: 38947 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 22:45:23.991102 21769 mvcc.cc:205] Tried to move back new op lower bound from 7333381423015616512 to 7333381422565175296. Current Snapshot: MvccSnapshot[applied={T|T < 7333381423015616512}]
I20260925 22:45:23.993258 21771 mvcc.cc:205] Tried to move back new op lower bound from 7333381423015616512 to 7333381422565175296. Current Snapshot: MvccSnapshot[applied={T|T < 7333381423015616512}]
I20260925 22:45:24.002105 21278 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54958:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 22:45:24.002339 21278 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54958:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 22:45:24.005070 21278 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f1830946663e4d1c89a857ffce4de755: Sending DeleteTablet for 3 replicas of tablet 0acde504c69947e991398abfd6fea6d2
I20260925 22:45:24.006129 21672 tablet_service.cc:1514] Processing DeleteTablet for tablet 0acde504c69947e991398abfd6fea6d2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 22:45:23 UTC) from {username='kudu', principal='kudu/127.10.198.62@KRBTEST.COM'} at 127.0.0.1:36422
I20260925 22:45:24.006167 21538 tablet_service.cc:1514] Processing DeleteTablet for tablet 0acde504c69947e991398abfd6fea6d2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 22:45:23 UTC) from {username='kudu', principal='kudu/127.10.198.62@KRBTEST.COM'} at 127.0.0.1:33830
I20260925 22:45:24.006695 21404 tablet_service.cc:1514] Processing DeleteTablet for tablet 0acde504c69947e991398abfd6fea6d2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 22:45:23 UTC) from {username='kudu', principal='kudu/127.10.198.62@KRBTEST.COM'} at 127.0.0.1:34052
I20260925 22:45:24.006747 21779 tablet_replica.cc:330] T 0acde504c69947e991398abfd6fea6d2 P 85e9146c658b4ab480aa7b5fffc53357: stopping tablet replica
I20260925 22:45:24.007200 21781 tablet_replica.cc:330] T 0acde504c69947e991398abfd6fea6d2 P 54d82bc61dea4af89266201d8a1deb47: stopping tablet replica
I20260925 22:45:24.007203 21779 raft_consensus.cc:2229] T 0acde504c69947e991398abfd6fea6d2 P 85e9146c658b4ab480aa7b5fffc53357 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 22:45:24.007485 21780 tablet_replica.cc:330] T 0acde504c69947e991398abfd6fea6d2 P 11aa0603d7954c32bf623cb954037c1c: stopping tablet replica
I20260925 22:45:24.007679 21781 raft_consensus.cc:2229] T 0acde504c69947e991398abfd6fea6d2 P 54d82bc61dea4af89266201d8a1deb47 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 22:45:24.007772 21780 raft_consensus.cc:2229] T 0acde504c69947e991398abfd6fea6d2 P 11aa0603d7954c32bf623cb954037c1c [term 1 LEADER]: Raft consensus shutting down.
I20260925 22:45:24.008093 11032 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrWXm8X/build/debug/bin/kudu with pid 21341
I20260925 22:45:24.008178 21780 raft_consensus.cc:2258] T 0acde504c69947e991398abfd6fea6d2 P 11aa0603d7954c32bf623cb954037c1c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 22:45:24.010329 21779 raft_consensus.cc:2258] T 0acde504c69947e991398abfd6fea6d2 P 85e9146c658b4ab480aa7b5fffc53357 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 22:45:24.012012 21780 ts_tablet_manager.cc:1905] T 0acde504c69947e991398abfd6fea6d2 P 11aa0603d7954c32bf623cb954037c1c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 22:45:24.012625 21779 ts_tablet_manager.cc:1905] T 0acde504c69947e991398abfd6fea6d2 P 85e9146c658b4ab480aa7b5fffc53357: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 22:45:24.015607 21779 ts_tablet_manager.cc:1918] T 0acde504c69947e991398abfd6fea6d2 P 85e9146c658b4ab480aa7b5fffc53357: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 22:45:24.015758 21779 log.cc:1196] T 0acde504c69947e991398abfd6fea6d2 P 85e9146c658b4ab480aa7b5fffc53357: Deleting WAL directory at /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790376259650642-11032-0/minicluster-data/ts-2/wal/wals/0acde504c69947e991398abfd6fea6d2
I20260925 22:45:24.016182 21779 ts_tablet_manager.cc:1939] T 0acde504c69947e991398abfd6fea6d2 P 85e9146c658b4ab480aa7b5fffc53357: Deleting consensus metadata
I20260925 22:45:24.016738 21780 ts_tablet_manager.cc:1918] T 0acde504c69947e991398abfd6fea6d2 P 11aa0603d7954c32bf623cb954037c1c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
W20260925 22:45:24.016721 21265 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.10.198.1:38947 (error 108)
I20260925 22:45:24.016949 21780 log.cc:1196] T 0acde504c69947e991398abfd6fea6d2 P 11aa0603d7954c32bf623cb954037c1c: Deleting WAL directory at /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790376259650642-11032-0/minicluster-data/ts-1/wal/wals/0acde504c69947e991398abfd6fea6d2
I20260925 22:45:24.017407 21780 ts_tablet_manager.cc:1939] T 0acde504c69947e991398abfd6fea6d2 P 11aa0603d7954c32bf623cb954037c1c: Deleting consensus metadata
I20260925 22:45:24.017624 11032 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrWXm8X/build/debug/bin/kudu with pid 21475
I20260925 22:45:24.017767 21266 catalog_manager.cc:4841] TS 85e9146c658b4ab480aa7b5fffc53357 (127.10.198.3:37315): tablet 0acde504c69947e991398abfd6fea6d2 (table test-table [id=1b0d2fb1d4754f59b3dc45e834573c60]) successfully deleted
I20260925 22:45:24.019148 21783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:45:24.018732 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.198.1:38947 (local address 127.0.0.1:34056)
0925 22:45:24.018962 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0925 22:45:24.019046 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.198.1:38947: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":87,"threads_started":1}
W20260925 22:45:24.019424 21265 catalog_manager.cc:4584] TS 54d82bc61dea4af89266201d8a1deb47 (127.10.198.1:38947): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 0acde504c69947e991398abfd6fea6d2: Network error: Client connection negotiation failed: client connection to 127.10.198.1:38947: connect: Connection refused (error 111)
I20260925 22:45:24.024439 11032 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrWXm8X/build/debug/bin/kudu with pid 21609
I20260925 22:45:24.026410 21783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:45:24.026072 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.198.2:46263 (local address 127.0.0.1:33850)
0925 22:45:24.026220 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0925 22:45:24.026294 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.198.2:46263: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260925 22:45:24.030751 11032 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrWXm8X/build/debug/bin/kudu with pid 21247
2026-09-25T22:45:24Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (2790 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376259650642-11032-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 22:45:24 dist-test-slave-5vdr krb5kdc[21786](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 22:45:24 dist-test-slave-5vdr krb5kdc[21786](info): set up 2 sockets
Sep 25 22:45:24 dist-test-slave-5vdr krb5kdc[21786](info): commencing operation
krb5kdc: starting...
W20260925 22:45:25.081277 11032 mini_kdc.cc:121] Time spent starting KDC: real 1.026s user 0.001s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376259650642-11032-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376259650642-11032-0/krb5kdc/test-user.keytab.
Sep 25 22:45:25 dist-test-slave-5vdr krb5kdc[21786](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376325, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T22:45:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T22:45:25Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.10.198.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.10.198.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.10.198.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376259650642-11032-0/krb5kdc/rangeradmin_127.10.198.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.10.198.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376259650642-11032-0/krb5kdc/rangeradmin_127.10.198.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.10.198.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.10.198.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.10.198.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376259650642-11032-0/krb5kdc/rangerlookup_127.10.198.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.10.198.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376259650642-11032-0/krb5kdc/rangerlookup_127.10.198.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.10.198.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.10.198.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.10.198.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376259650642-11032-0/krb5kdc/HTTP_127.10.198.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.10.198.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376259650642-11032-0/krb5kdc/HTTP_127.10.198.20@KRBTEST.COM.keytab.
I20260925 22:45:25.267123 11032 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376259650642-11032-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskrWXm8X/build/debug/bin/postgres/pg_ctl -D /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376259650642-11032-0/minicluster-data/postgres -l logfile start
2026-09-25 22:45:27.972 UTC [21816] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-25 22:45:27.972 UTC [21816] LOG: listening on IPv4 address "127.10.198.20", port 46661
2026-09-25 22:45:27.976 UTC [21816] LOG: listening on Unix socket "/tmp/.s.PGSQL.46661"
2026-09-25 22:45:28.012 UTC [21819] LOG: database system was shut down at 2026-09-25 22:45:26 UTC
2026-09-25 22:45:28.017 UTC [21816] LOG: database system is ready to accept connections
I20260925 22:45:28.972853 11032 mini_postgres.cc:96] Postgres bound to 46661
2026-09-25 22:45:28.980 UTC [21827] FATAL: database "slave" does not exist
127.10.198.20:46661 - accepting connections
I20260925 22:45:28.980969 11032 mini_ranger.cc:161] Starting Ranger...
I20260925 22:45:28.997476 11032 mini_ranger.cc:84] Created miniranger Postgres user
I20260925 22:45:29.523001 11032 mini_ranger.cc:87] Created ranger Postgres database
I20260925 22:45:29.523118 11032 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376259650642-11032-0/minicluster-data/ranger-admin
2026-09-25 22:45:29,813 [I] DB FLAVOR :POSTGRES
2026-09-25 22:45:29,814 [I] --------- Verifying Ranger DB connection ---------
2026-09-25 22:45:29,814 [I] Checking connection..
2026-09-25 22:45:29,814 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskrWXm8X/build/debug/bin/postgresql.jar:/tmp/dist-test-taskrWXm8X/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.198.20:46661/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-25 22:45:30,094 [I] Checking connection passed.
2026-09-25 22:45:30,095 [I] --------- Verifying version history table ---------
2026-09-25 22:45:30,095 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskrWXm8X/build/debug/bin/postgresql.jar:/tmp/dist-test-taskrWXm8X/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.198.20:46661/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-25 22:45:30,374 [I] Table x_db_version_h does not exist in database ranger
2026-09-25 22:45:30,375 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskrWXm8X/build/debug/bin/postgresql.jar:/tmp/dist-test-taskrWXm8X/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.198.20:46661/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-25 22:45:30,654 [I] Table x_db_version_h does not exist in database ranger
2026-09-25 22:45:30,654 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-25 22:45:30,654 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskrWXm8X/build/debug/bin/postgresql.jar:/tmp/dist-test-taskrWXm8X/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.198.20:46661/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskrWXm8X/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-25 22:45:30.928 UTC [21891] WARNING: there is no transaction in progress
2026-09-25 22:45:30,935 [I] create_dbversion_catalog.sql file imported successfully
2026-09-25 22:45:30,935 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskrWXm8X/build/debug/bin/postgresql.jar:/tmp/dist-test-taskrWXm8X/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.198.20:46661/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-25 22:45:31,198 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-25 22:45:31,198 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-25 22:45:31,198 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskrWXm8X/build/debug/bin/postgresql.jar:/tmp/dist-test-taskrWXm8X/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.198.20:46661/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-25 22:45:31,477 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskrWXm8X/build/debug/bin/postgresql.jar:/tmp/dist-test-taskrWXm8X/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.198.20:46661/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-25 22:45:31,742 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskrWXm8X/build/debug/bin/postgresql.jar:/tmp/dist-test-taskrWXm8X/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.198.20:46661/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-5vdr.c.gcp-upstream.internal','N') ;"
2026-09-25 22:45:31,980 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskrWXm8X/build/debug/bin/postgresql.jar:/tmp/dist-test-taskrWXm8X/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.198.20:46661/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-25 22:45:32,239 [I] Table x_portal_user does not exist in database ranger
2026-09-25 22:45:32,240 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskrWXm8X/build/debug/bin/postgresql.jar:/tmp/dist-test-taskrWXm8X/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.198.20:46661/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-25 22:45:32,492 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-25 22:45:32,492 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskrWXm8X/build/debug/bin/postgresql.jar:/tmp/dist-test-taskrWXm8X/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.198.20:46661/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-25 22:45:32,749 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskrWXm8X/build/debug/bin/postgresql.jar:/tmp/dist-test-taskrWXm8X/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.198.20:46661/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-25 22:45:32,995 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-25 22:45:32,995 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskrWXm8X/build/debug/bin/postgresql.jar:/tmp/dist-test-taskrWXm8X/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.198.20:46661/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskrWXm8X/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-25 22:45:34.107 UTC [21999] WARNING: there is no transaction in progress
2026-09-25 22:45:34.127 UTC [21999] WARNING: there is no transaction in progress
2026-09-25 22:45:34.147 UTC [21999] WARNING: there is no transaction in progress
2026-09-25 22:45:34.165 UTC [21999] WARNING: there is no transaction in progress
2026-09-25 22:45:34.183 UTC [21999] WARNING: there is no transaction in progress
2026-09-25 22:45:34.203 UTC [21999] WARNING: there is no transaction in progress
2026-09-25 22:45:34.276 UTC [21999] WARNING: there is no transaction in progress
2026-09-25 22:45:34.285 UTC [21999] WARNING: there is no transaction in progress
2026-09-25 22:45:34.302 UTC [21999] WARNING: there is no transaction in progress
2026-09-25 22:45:34.314 UTC [21999] WARNING: there is no transaction in progress
2026-09-25 22:45:34.326 UTC [21999] WARNING: there is no transaction in progress
2026-09-25 22:45:34.341 UTC [21999] WARNING: there is no transaction in progress
2026-09-25 22:45:34.353 UTC [21999] WARNING: there is no transaction in progress
2026-09-25 22:45:34.359 UTC [21999] WARNING: there is no transaction in progress
2026-09-25 22:45:34.367 UTC [21999] WARNING: there is no transaction in progress
2026-09-25 22:45:34.375 UTC [21999] WARNING: there is no transaction in progress
2026-09-25 22:45:35,144 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-25 22:45:35,145 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskrWXm8X/build/debug/bin/postgresql.jar:/tmp/dist-test-taskrWXm8X/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.198.20:46661/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-25 22:45:35,424 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-25 22:45:35,425 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskrWXm8X/build/debug/bin/postgresql.jar:/tmp/dist-test-taskrWXm8X/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.198.20:46661/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-25 22:45:35,702 [I] Table x_portal_user already exists in database 'ranger'
2026-09-25 22:45:35,702 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskrWXm8X/build/debug/bin/postgresql.jar:/tmp/dist-test-taskrWXm8X/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.198.20:46661/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-25 22:45:35,972 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-25 22:45:35,973 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskrWXm8X/build/debug/bin/postgresql.jar:/tmp/dist-test-taskrWXm8X/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.198.20:46661/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-25 22:45:36,220 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskrWXm8X/build/debug/bin/postgresql.jar:/tmp/dist-test-taskrWXm8X/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.198.20:46661/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-25 22:45:36,484 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskrWXm8X/build/debug/bin/postgresql.jar:/tmp/dist-test-taskrWXm8X/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.198.20:46661/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-5vdr.c.gcp-upstream.internal';"
2026-09-25 22:45:36,724 [I] CORE_DB_SCHEMA import status has been updated
2026-09-25 22:45:36,725 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskrWXm8X/build/debug/bin/postgresql.jar:/tmp/dist-test-taskrWXm8X/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.198.20:46661/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-25 22:45:37,006 [I] DB_PATCHES have already been applied
I20260925 22:45:37.013240 11032 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376259650642-11032-0/minicluster-data/ranger-admin:/tmp/dist-test-taskrWXm8X/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskrWXm8X/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskrWXm8X/build/debug/bin/postgresql.jar:/tmp/dist-test-taskrWXm8X/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260925 22:45:37.013345 11032 mini_ranger.cc:193] Using host: 127.10.198.20
I20260925 22:45:37.017226 11032 mini_ranger.cc:227] Ranger admin URL: http://127.10.198.20:53111
Sep 25, 2026 10:45:37 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 25, 2026 10:45:37 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 25, 2026 10:45:37 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376259650642-11032-0/minicluster-data/ranger-admin/ews/webapp
Sep 25, 2026 10:45:37 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376259650642-11032-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 25, 2026 10:45:37 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376259650642-11032-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 25, 2026 10:45:37 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376259650642-11032-0/minicluster-data/ranger-admin/ews/webapp].
Sep 25, 2026 10:45:37 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260925 22:45:38.176532 11032 mini_ranger.cc:160] Time spent starting Ranger: real 9.196s user 0.008s sys 0.003s
Sep 25, 2026 10:46:00 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 25, 2026 10:46:00 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Sep 25, 2026 10:46:00 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 25, 2026 10:46:00 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 25, 2026 10:46:00 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 25, 2026 10:46:00 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 25, 2026 10:46:00 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 25, 2026 10:46:00 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 25, 2026 10:46:00 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 25, 2026 10:46:00 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 25, 2026 10:46:00 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 25, 2026 10:46:00 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 25, 2026 10:46:00 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 25, 2026 10:46:00 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 25, 2026 10:46:00 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 25, 2026 10:46:00 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 25, 2026 10:46:00 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 25, 2026 10:46:01 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260925 22:46:02.591339 11032 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.10.198.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.10.198.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.10.198.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376259650642-11032-0/krb5kdc/rangerkms_127.10.198.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.10.198.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376259650642-11032-0/krb5kdc/rangerkms_127.10.198.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.10.198.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.10.198.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.10.198.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376259650642-11032-0/krb5kdc/HTTP_127.10.198.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.10.198.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376259650642-11032-0/krb5kdc/HTTP_127.10.198.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 25 22:46:02 dist-test-slave-5vdr krb5kdc[21786](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376362, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260925 22:46:02.698518 11032 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260925 22:46:02.709497 11032 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260925 22:46:03.816510 11032 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260925 22:46:03.816630 11032 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376259650642-11032-0/minicluster-data/ranger-kms
I20260925 22:46:03.816762 11032 mini_ranger_kms.cc:227] Using postgres at 127.10.198.20:46661
2026-09-25 22:46:04,085 [I] DB FLAVOR :POSTGRES
2026-09-25 22:46:04,085 [I] --------- Verifying Ranger DB connection ---------
2026-09-25 22:46:04,085 [I] Checking connection
2026-09-25 22:46:04,085 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskrWXm8X/build/debug/bin/postgresql.jar:/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376259650642-11032-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.198.20:46661/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-25 22:46:04,404 [I] connection success
2026-09-25 22:46:04,404 [I] --------- Verifying Ranger DB tables ---------
2026-09-25 22:46:04,404 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-25 22:46:04,405 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskrWXm8X/build/debug/bin/postgresql.jar:/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376259650642-11032-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.198.20:46661/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-25 22:46:04,673 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-25 22:46:04,674 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-25 22:46:04,674 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-25 22:46:04,674 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskrWXm8X/build/debug/bin/postgresql.jar:/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376259650642-11032-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.198.20:46661/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376259650642-11032-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-25 22:46:04,986 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260925 22:46:05.558501 11032 mini_ranger_kms.cc:310] Created kms service
I20260925 22:46:05.772840 11032 mini_ranger_kms.cc:326] Created kudu user
I20260925 22:46:05.821559 11032 mini_ranger_kms.cc:343] Created rangerkms user
Sep 25 22:46:05 dist-test-slave-5vdr krb5kdc[21786](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376362, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.10.198.20@KRBTEST.COM
I20260925 22:46:06.288935 11032 mini_ranger_kms.cc:384] Added ranger policy
I20260925 22:46:06.289117 11032 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376259650642-11032-0/minicluster-data/ranger-kms:/tmp/dist-test-taskrWXm8X/build/debug/bin/postgresql.jar:/tmp/dist-test-taskrWXm8X/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskrWXm8X/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260925 22:46:06.289142 11032 mini_ranger_kms.cc:240] Using host: 127.10.198.20
I20260925 22:46:06.292968 11032 mini_ranger_kms.cc:276] Ranger KMS PID: 22180
I20260925 22:46:06.293156 11032 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.10.198.20:55509
Sep 25, 2026 10:46:07 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 25, 2026 10:46:07 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 25, 2026 10:46:07 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376259650642-11032-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 25, 2026 10:46:07 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376259650642-11032-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 25, 2026 10:46:07 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376259650642-11032-0/minicluster-data/ranger-kms/ews/webapp].
Sep 25, 2026 10:46:07 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260925 22:46:08.555485 11032 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.857s user 0.018s sys 0.232s
I20260925 22:46:08.555645 11032 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260925 22:46:08.555784 11032 mini_ranger_kms.cc:401] 127.10.198.20:55509/kms/v1/keys
Sep 25 22:46:08 dist-test-slave-5vdr krb5kdc[21786](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790376368, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.10.198.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 25 22:46:09 dist-test-slave-5vdr krb5kdc[21786](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376368, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.10.198.20@KRBTEST.COM for HTTP/127.10.198.20@KRBTEST.COM
Sep 25 22:46:09 dist-test-slave-5vdr krb5kdc[21786](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376368, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.10.198.20@KRBTEST.COM for HTTP/127.10.198.20@KRBTEST.COM
Sep 25 22:46:12 dist-test-slave-5vdr krb5kdc[21786](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376372, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.10.198.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.198.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.198.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376259650642-11032-0/krb5kdc/kudu_127.10.198.62.keytab.
Entry for principal kudu/127.10.198.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376259650642-11032-0/krb5kdc/kudu_127.10.198.62.keytab.
I20260925 22:46:12.396101 11032 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrWXm8X/build/debug/bin/kudu
/tmp/dist-test-taskrWXm8X/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376259650642-11032-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376259650642-11032-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376259650642-11032-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376259650642-11032-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.198.62:40283
--webserver_interface=127.10.198.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376259650642-11032-0/krb5kdc/kudu_127.10.198.62.keytab
--principal=kudu/127.10.198.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.198.20:44819
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.198.62:40283
--ranger_config_path=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376259650642-11032-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.10.198.20:55509/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376259650642-11032-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376259650642-11032-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376259650642-11032-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:46:12.526309 22230 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:46:12.526691 22230 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:46:12.526796 22230 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:46:12.531358 22230 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 22:46:12.531482 22230 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:46:12.531522 22230 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:46:12.531574 22230 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 22:46:12.531639 22230 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 22:46:12.538074 22230 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.198.20:44819
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376259650642-11032-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376259650642-11032-0/minicluster-data/master-0/wal
--ranger_kms_url=127.10.198.20:55509/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.10.198.62:40283
--ranger_config_path=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376259650642-11032-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376259650642-11032-0/krb5kdc/kudu_127.10.198.62.keytab
--principal=kudu/127.10.198.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.198.62:40283
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376259650642-11032-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.198.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22230
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376259650642-11032-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:09:16 UTC on ccycloud.quasar-afmhyw.root.comops.site
build id 4673379
I20260925 22:46:12.539422 22230 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:46:12.540653 22230 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:46:12.552299 22235 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:46:12.555800 22238 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:46:12.556213 22236 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:46:12.557408 22230 server_base.cc:1034] running on GCE node
I20260925 22:46:12.558305 22230 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:46:12.560016 22230 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:46:12.561269 22230 hybrid_clock.cc:648] HybridClock initialized: now 1790376372561244 us; error 57 us; skew 500 ppm
Sep 25 22:46:12 dist-test-slave-5vdr krb5kdc[21786](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376372, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.198.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:46:12.570353 22230 init.cc:377] Logged in from keytab as kudu/127.10.198.62@KRBTEST.COM (short username kudu)
I20260925 22:46:12.572388 22230 webserver.cc:460] Webserver started at http://127.10.198.62:36317/ using document root <none> and password file <none>
I20260925 22:46:12.573269 22230 fs_manager.cc:359] Metadata directory not provided
I20260925 22:46:12.573392 22230 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:46:12.573693 22230 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 25 22:46:12 dist-test-slave-5vdr krb5kdc[21786](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376372, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.198.62@KRBTEST.COM for HTTP/127.10.198.20@KRBTEST.COM
I20260925 22:46:12.656948 22230 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376259650642-11032-0/minicluster-data/master-0/data/instance:
uuid: "68f68a9d8f944d0fbc86369d1685d646"
format_stamp: "Formatted at 2026-09-25 22:46:12 on dist-test-slave-5vdr"
server_key: "0603174d170058452ce6f2103ee6c371"
server_key_iv: "44b3bc550f55280e6d3024e8e82a2163"
server_key_version: "kuduclusterkey@0"
I20260925 22:46:12.657856 22230 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376259650642-11032-0/minicluster-data/master-0/wal/instance:
uuid: "68f68a9d8f944d0fbc86369d1685d646"
format_stamp: "Formatted at 2026-09-25 22:46:12 on dist-test-slave-5vdr"
server_key: "0603174d170058452ce6f2103ee6c371"
server_key_iv: "44b3bc550f55280e6d3024e8e82a2163"
server_key_version: "kuduclusterkey@0"
I20260925 22:46:12.664330 22230 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.002s sys 0.004s
I20260925 22:46:12.684034 22245 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:46:12.686053 22230 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260925 22:46:12.686267 22230 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376259650642-11032-0/minicluster-data/master-0/data,/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376259650642-11032-0/minicluster-data/master-0/wal
uuid: "68f68a9d8f944d0fbc86369d1685d646"
format_stamp: "Formatted at 2026-09-25 22:46:12 on dist-test-slave-5vdr"
server_key: "0603174d170058452ce6f2103ee6c371"
server_key_iv: "44b3bc550f55280e6d3024e8e82a2163"
server_key_version: "kuduclusterkey@0"
I20260925 22:46:12.686461 22230 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376259650642-11032-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376259650642-11032-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376259650642-11032-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:46:12.706161 22230 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:46:12.710328 22230 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:46:12.710606 22230 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:46:12.720950 22230 rpc_server.cc:307] RPC server started. Bound to: 127.10.198.62:40283
I20260925 22:46:12.720995 22297 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.198.62:40283 every 8 connection(s)
I20260925 22:46:12.722863 22230 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376259650642-11032-0/minicluster-data/master-0/data/info.pb
I20260925 22:46:12.726411 22298 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:46:12.727599 11032 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrWXm8X/build/debug/bin/kudu as pid 22230
I20260925 22:46:12.727742 11032 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376259650642-11032-0/minicluster-data/master-0/wal/instance
I20260925 22:46:12.728098 11032 external_mini_cluster.cc:1428] Setting key 2c293d673d2a726f06ccd83a14cce95b
I20260925 22:46:12.735001 22298 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 68f68a9d8f944d0fbc86369d1685d646: Bootstrap starting.
I20260925 22:46:12.738092 22298 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 68f68a9d8f944d0fbc86369d1685d646: Neither blocks nor log segments found. Creating new log.
I20260925 22:46:12.739259 22298 log.cc:824] T 00000000000000000000000000000000 P 68f68a9d8f944d0fbc86369d1685d646: Log is configured to *not* fsync() on all Append() calls
Sep 25 22:46:12 dist-test-slave-5vdr krb5kdc[21786](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376372, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.198.62@KRBTEST.COM
I20260925 22:46:12.742168 22298 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 68f68a9d8f944d0fbc86369d1685d646: No bootstrap required, opened a new log
I20260925 22:46:12.746061 22298 raft_consensus.cc:348] T 00000000000000000000000000000000 P 68f68a9d8f944d0fbc86369d1685d646 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "68f68a9d8f944d0fbc86369d1685d646" member_type: VOTER last_known_addr { host: "127.10.198.62" port: 40283 } }
I20260925 22:46:12.746522 22298 raft_consensus.cc:374] T 00000000000000000000000000000000 P 68f68a9d8f944d0fbc86369d1685d646 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:46:12.746698 22298 raft_consensus.cc:729] T 00000000000000000000000000000000 P 68f68a9d8f944d0fbc86369d1685d646 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 68f68a9d8f944d0fbc86369d1685d646, State: Initialized, Role: FOLLOWER
I20260925 22:46:12.747377 22298 consensus_queue.cc:260] T 00000000000000000000000000000000 P 68f68a9d8f944d0fbc86369d1685d646 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "68f68a9d8f944d0fbc86369d1685d646" member_type: VOTER last_known_addr { host: "127.10.198.62" port: 40283 } }
I20260925 22:46:12.748294 22301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:46:12.729769 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44814 (local address 127.10.198.62:40283)
0925 22:46:12.730837 (+ 1068us) server_negotiation.cc:206] Beginning negotiation
0925 22:46:12.730860 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0925 22:46:12.730927 (+ 67us) server_negotiation.cc:407] Connection header received
0925 22:46:12.731991 (+ 1064us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:46:12.732021 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:46:12.732543 (+ 522us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:46:12.732998 (+ 455us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:46:12.733982 (+ 984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:46:12.735617 (+ 1635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:46:12.736980 (+ 1363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:46:12.737788 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:46:12.737995 (+ 207us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:46:12.741271 (+ 3276us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:46:12.741279 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:46:12.741302 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:46:12.741341 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:46:12.744397 (+ 3056us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:46:12.745091 (+ 694us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:46:12.745098 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:46:12.745107 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:46:12.745274 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:46:12.745632 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:46:12.745635 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:46:12.745637 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:46:12.746222 (+ 585us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:46:12.746446 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0925 22:46:12.746826 (+ 380us) server_negotiation.cc:299] Negotiation successful
0925 22:46:12.747170 (+ 344us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":861,"thread_start_us":617,"threads_started":1}
I20260925 22:46:12.747568 22298 raft_consensus.cc:388] T 00000000000000000000000000000000 P 68f68a9d8f944d0fbc86369d1685d646 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 22:46:12.748731 22298 raft_consensus.cc:482] T 00000000000000000000000000000000 P 68f68a9d8f944d0fbc86369d1685d646 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 22:46:12.748858 22298 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 68f68a9d8f944d0fbc86369d1685d646 [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:46:12.750116 22298 raft_consensus.cc:504] T 00000000000000000000000000000000 P 68f68a9d8f944d0fbc86369d1685d646 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "68f68a9d8f944d0fbc86369d1685d646" member_type: VOTER last_known_addr { host: "127.10.198.62" port: 40283 } }
I20260925 22:46:12.750574 22298 leader_election.cc:302] T 00000000000000000000000000000000 P 68f68a9d8f944d0fbc86369d1685d646 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 68f68a9d8f944d0fbc86369d1685d646; no voters:
I20260925 22:46:12.750993 22298 leader_election.cc:288] T 00000000000000000000000000000000 P 68f68a9d8f944d0fbc86369d1685d646 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 22:46:12.751438 22303 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 68f68a9d8f944d0fbc86369d1685d646 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 22:46:12.751643 22303 raft_consensus.cc:686] T 00000000000000000000000000000000 P 68f68a9d8f944d0fbc86369d1685d646 [term 1 LEADER]: Becoming Leader. State: Replica: 68f68a9d8f944d0fbc86369d1685d646, State: Running, Role: LEADER
I20260925 22:46:12.752027 22303 consensus_queue.cc:237] T 00000000000000000000000000000000 P 68f68a9d8f944d0fbc86369d1685d646 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "68f68a9d8f944d0fbc86369d1685d646" member_type: VOTER last_known_addr { host: "127.10.198.62" port: 40283 } }
I20260925 22:46:12.752562 22298 sys_catalog.cc:564] T 00000000000000000000000000000000 P 68f68a9d8f944d0fbc86369d1685d646 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 22:46:12.753885 22304 sys_catalog.cc:455] T 00000000000000000000000000000000 P 68f68a9d8f944d0fbc86369d1685d646 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "68f68a9d8f944d0fbc86369d1685d646" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "68f68a9d8f944d0fbc86369d1685d646" member_type: VOTER last_known_addr { host: "127.10.198.62" port: 40283 } } }
I20260925 22:46:12.754055 22304 sys_catalog.cc:458] T 00000000000000000000000000000000 P 68f68a9d8f944d0fbc86369d1685d646 [sys.catalog]: This master's current role is: LEADER
I20260925 22:46:12.754721 22305 sys_catalog.cc:455] T 00000000000000000000000000000000 P 68f68a9d8f944d0fbc86369d1685d646 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 68f68a9d8f944d0fbc86369d1685d646. Latest consensus state: current_term: 1 leader_uuid: "68f68a9d8f944d0fbc86369d1685d646" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "68f68a9d8f944d0fbc86369d1685d646" member_type: VOTER last_known_addr { host: "127.10.198.62" port: 40283 } } }
I20260925 22:46:12.754848 22305 sys_catalog.cc:458] T 00000000000000000000000000000000 P 68f68a9d8f944d0fbc86369d1685d646 [sys.catalog]: This master's current role is: LEADER
I20260925 22:46:12.754707 22312 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 22:46:12.755128 22298 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376259650642-11032-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260925 22:46:12.763634 22312 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 22:46:12.775733 22312 catalog_manager.cc:1347] Generated new cluster ID: 916e89de6b1f4c1da3e737398ee7ffa2
I20260925 22:46:12.775863 22312 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 22:46:12.788079 22312 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 22:46:12.790680 22312 catalog_manager.cc:1504] Loading token signing keys...
I20260925 22:46:12.804862 22312 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 68f68a9d8f944d0fbc86369d1685d646: Generated new TSK 0
I20260925 22:46:12.805871 22312 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.10.198.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.198.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.198.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376259650642-11032-0/krb5kdc/kudu_127.10.198.1.keytab.
Entry for principal kudu/127.10.198.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376259650642-11032-0/krb5kdc/kudu_127.10.198.1.keytab.
I20260925 22:46:13.636207 11032 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrWXm8X/build/debug/bin/kudu
/tmp/dist-test-taskrWXm8X/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376259650642-11032-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376259650642-11032-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376259650642-11032-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376259650642-11032-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.10.198.20:55509/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.198.1:0
--local_ip_for_outbound_sockets=127.10.198.1
--webserver_interface=127.10.198.1
--webserver_port=0
--tserver_master_addrs=127.10.198.62:40283
--keytab_file=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376259650642-11032-0/krb5kdc/kudu_127.10.198.1.keytab
--principal=kudu/127.10.198.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.198.20:44819
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376259650642-11032-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376259650642-11032-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376259650642-11032-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:46:13.786568 22343 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:46:13.787065 22343 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:46:13.787177 22343 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:46:13.792493 22343 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:46:13.792575 22343 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:46:13.792644 22343 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.198.1
I20260925 22:46:13.800192 22343 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.198.20:44819
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376259650642-11032-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376259650642-11032-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.10.198.20:55509/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376259650642-11032-0/krb5kdc/kudu_127.10.198.1.keytab
--principal=kudu/127.10.198.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.198.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376259650642-11032-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.198.1
--webserver_port=0
--tserver_master_addrs=127.10.198.62:40283
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22343
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.198.1
--log_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376259650642-11032-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:09:16 UTC on ccycloud.quasar-afmhyw.root.comops.site
build id 4673379
I20260925 22:46:13.801716 22343 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:46:13.803141 22343 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:46:13.815243 22351 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:46:13.815411 22353 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:46:13.815241 22350 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:46:13.816111 22343 server_base.cc:1034] running on GCE node
I20260925 22:46:13.816676 22343 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:46:13.817634 22343 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:46:13.818887 22343 hybrid_clock.cc:648] HybridClock initialized: now 1790376373818864 us; error 46 us; skew 500 ppm
Sep 25 22:46:13 dist-test-slave-5vdr krb5kdc[21786](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376373, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.198.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:46:13.823119 22343 init.cc:377] Logged in from keytab as kudu/127.10.198.1@KRBTEST.COM (short username kudu)
I20260925 22:46:13.824476 22343 webserver.cc:460] Webserver started at http://127.10.198.1:33765/ using document root <none> and password file <none>
I20260925 22:46:13.825248 22343 fs_manager.cc:359] Metadata directory not provided
I20260925 22:46:13.825356 22343 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:46:13.825649 22343 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 25 22:46:13 dist-test-slave-5vdr krb5kdc[21786](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376373, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.198.1@KRBTEST.COM for HTTP/127.10.198.20@KRBTEST.COM
I20260925 22:46:13.845250 22343 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376259650642-11032-0/minicluster-data/ts-0/data/instance:
uuid: "6972169c9517492eaf91bc5ec8e851b7"
format_stamp: "Formatted at 2026-09-25 22:46:13 on dist-test-slave-5vdr"
server_key: "04bd14d5dd29d157367fcb14f41d7ca9"
server_key_iv: "81a9011a44563718b3589fd7d3b364ca"
server_key_version: "kuduclusterkey@0"
I20260925 22:46:13.846216 22343 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376259650642-11032-0/minicluster-data/ts-0/wal/instance:
uuid: "6972169c9517492eaf91bc5ec8e851b7"
format_stamp: "Formatted at 2026-09-25 22:46:13 on dist-test-slave-5vdr"
server_key: "04bd14d5dd29d157367fcb14f41d7ca9"
server_key_iv: "81a9011a44563718b3589fd7d3b364ca"
server_key_version: "kuduclusterkey@0"
I20260925 22:46:13.851330 22343 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260925 22:46:13.892362 22360 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:46:13.894474 22343 fs_manager.cc:725] Time spent opening block manager: real 0.020s user 0.003s sys 0.001s
I20260925 22:46:13.894670 22343 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376259650642-11032-0/minicluster-data/ts-0/data,/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376259650642-11032-0/minicluster-data/ts-0/wal
uuid: "6972169c9517492eaf91bc5ec8e851b7"
format_stamp: "Formatted at 2026-09-25 22:46:13 on dist-test-slave-5vdr"
server_key: "04bd14d5dd29d157367fcb14f41d7ca9"
server_key_iv: "81a9011a44563718b3589fd7d3b364ca"
server_key_version: "kuduclusterkey@0"
I20260925 22:46:13.894856 22343 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376259650642-11032-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376259650642-11032-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376259650642-11032-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:46:13.913578 22343 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:46:13.917972 22343 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:46:13.918241 22343 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:46:13.919028 22343 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 22:46:13.920239 22343 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:46:13.920315 22343 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:46:13.920380 22343 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:46:13.920409 22343 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 22:46:13.933876 22343 rpc_server.cc:307] RPC server started. Bound to: 127.10.198.1:36929
I20260925 22:46:13.935290 22343 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376259650642-11032-0/minicluster-data/ts-0/data/info.pb
I20260925 22:46:13.936966 22475 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.198.1:36929 every 8 connection(s)
I20260925 22:46:13.939692 11032 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrWXm8X/build/debug/bin/kudu as pid 22343
I20260925 22:46:13.939828 11032 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376259650642-11032-0/minicluster-data/ts-0/wal/instance
I20260925 22:46:13.940188 11032 external_mini_cluster.cc:1428] Setting key 2e973efff703fb7d1c55e13ede375683
Sep 25 22:46:13 dist-test-slave-5vdr krb5kdc[21786](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376373, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.198.1@KRBTEST.COM for kudu/127.10.198.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.10.198.2@KRBTEST.COM; defaulting to no policy
I20260925 22:46:13.959154 22478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:46:13.939483 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.198.1:38925 (local address 127.10.198.62:40283)
0925 22:46:13.941407 (+ 1924us) server_negotiation.cc:206] Beginning negotiation
0925 22:46:13.941411 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:46:13.943475 (+ 2064us) server_negotiation.cc:407] Connection header received
0925 22:46:13.943537 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:46:13.943543 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:46:13.943620 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:46:13.943718 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:46:13.945571 (+ 1853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:46:13.946499 (+ 928us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:46:13.948685 (+ 2186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:46:13.949071 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:46:13.949193 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:46:13.952888 (+ 3695us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:46:13.952893 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:46:13.952895 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:46:13.952931 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:46:13.955518 (+ 2587us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:46:13.956570 (+ 1052us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:46:13.956574 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:46:13.956576 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:46:13.956692 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:46:13.957160 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:46:13.957163 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:46:13.957165 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:46:13.957413 (+ 248us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:46:13.957504 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0925 22:46:13.958788 (+ 1284us) server_negotiation.cc:299] Negotiation successful
0925 22:46:13.958957 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1829,"thread_start_us":110,"threads_started":1}
I20260925 22:46:13.959944 22479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:46:13.940094 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.198.62:40283 (local address 127.10.198.1:38925)
0925 22:46:13.941715 (+ 1621us) negotiation.cc:107] Waiting for socket to connect
0925 22:46:13.941800 (+ 85us) client_negotiation.cc:174] Beginning negotiation
0925 22:46:13.943080 (+ 1280us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:46:13.944026 (+ 946us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:46:13.944039 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:46:13.944476 (+ 437us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:46:13.945282 (+ 806us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:46:13.945308 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:46:13.946734 (+ 1426us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:46:13.946739 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:46:13.948426 (+ 1687us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:46:13.948448 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:46:13.949436 (+ 988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:46:13.949440 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:46:13.949690 (+ 250us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:46:13.950522 (+ 832us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:46:13.950548 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:46:13.952601 (+ 2053us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:46:13.955733 (+ 3132us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:46:13.955741 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:46:13.955765 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:46:13.956405 (+ 640us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:46:13.956813 (+ 408us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:46:13.956817 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:46:13.956819 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:46:13.957045 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:46:13.957562 (+ 517us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:46:13.957570 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:46:13.957955 (+ 385us) client_negotiation.cc:769] Sending connection context
0925 22:46:13.958195 (+ 240us) client_negotiation.cc:240] Negotiation successful
0925 22:46:13.958637 (+ 442us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1368,"thread_start_us":166,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.198.2@KRBTEST.COM" created.
I20260925 22:46:13.961709 22476 heartbeater.cc:347] Connected to a master server at 127.10.198.62:40283
I20260925 22:46:13.962008 22476 heartbeater.cc:464] Registering TS with master...
I20260925 22:46:13.962687 22476 heartbeater.cc:511] Master 127.10.198.62:40283 requested a full tablet report, sending...
I20260925 22:46:13.964800 22262 ts_manager.cc:194] Registered new tserver with Master: 6972169c9517492eaf91bc5ec8e851b7 (127.10.198.1:36929)
I20260925 22:46:13.966500 22262 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.198.1@KRBTEST.COM'} at 127.10.198.1:38925
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.198.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376259650642-11032-0/krb5kdc/kudu_127.10.198.2.keytab.
Entry for principal kudu/127.10.198.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376259650642-11032-0/krb5kdc/kudu_127.10.198.2.keytab.
I20260925 22:46:13.983091 11032 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrWXm8X/build/debug/bin/kudu
/tmp/dist-test-taskrWXm8X/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376259650642-11032-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376259650642-11032-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376259650642-11032-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376259650642-11032-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.10.198.20:55509/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.198.2:0
--local_ip_for_outbound_sockets=127.10.198.2
--webserver_interface=127.10.198.2
--webserver_port=0
--tserver_master_addrs=127.10.198.62:40283
--keytab_file=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376259650642-11032-0/krb5kdc/kudu_127.10.198.2.keytab
--principal=kudu/127.10.198.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.198.20:44819
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376259650642-11032-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376259650642-11032-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376259650642-11032-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 25 22:46:14 dist-test-slave-5vdr krb5kdc[21786](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790376374, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.198.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260925 22:46:14.143355 22482 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:46:14.143743 22482 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:46:14.143913 22482 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:46:14.149678 22482 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:46:14.149833 22482 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:46:14.149956 22482 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.198.2
I20260925 22:46:14.159065 22482 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.198.20:44819
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376259650642-11032-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376259650642-11032-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.10.198.20:55509/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376259650642-11032-0/krb5kdc/kudu_127.10.198.2.keytab
--principal=kudu/127.10.198.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.198.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376259650642-11032-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.198.2
--webserver_port=0
--tserver_master_addrs=127.10.198.62:40283
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22482
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.198.2
--log_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376259650642-11032-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:09:16 UTC on ccycloud.quasar-afmhyw.root.comops.site
build id 4673379
I20260925 22:46:14.160755 22482 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:46:14.162523 22482 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:46:14.175946 22489 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:46:14.175949 22488 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:46:14.176229 22491 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:46:14.177842 22482 server_base.cc:1034] running on GCE node
I20260925 22:46:14.178396 22482 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:46:14.179382 22482 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:46:14.180608 22482 hybrid_clock.cc:648] HybridClock initialized: now 1790376374180556 us; error 74 us; skew 500 ppm
Sep 25 22:46:14 dist-test-slave-5vdr krb5kdc[21786](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376374, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.198.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:46:14.185038 22482 init.cc:377] Logged in from keytab as kudu/127.10.198.2@KRBTEST.COM (short username kudu)
I20260925 22:46:14.186318 22482 webserver.cc:460] Webserver started at http://127.10.198.2:40817/ using document root <none> and password file <none>
I20260925 22:46:14.187067 22482 fs_manager.cc:359] Metadata directory not provided
I20260925 22:46:14.187183 22482 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:46:14.187464 22482 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 25 22:46:14 dist-test-slave-5vdr krb5kdc[21786](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376374, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.198.2@KRBTEST.COM for HTTP/127.10.198.20@KRBTEST.COM
I20260925 22:46:14.239784 22482 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376259650642-11032-0/minicluster-data/ts-1/data/instance:
uuid: "176c4879d00447ecafb59a251a267394"
format_stamp: "Formatted at 2026-09-25 22:46:14 on dist-test-slave-5vdr"
server_key: "02f43aa0830ab9e1a0f06557231be321"
server_key_iv: "8fd08ebab212fda8549fa42e70350721"
server_key_version: "kuduclusterkey@0"
I20260925 22:46:14.240794 22482 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376259650642-11032-0/minicluster-data/ts-1/wal/instance:
uuid: "176c4879d00447ecafb59a251a267394"
format_stamp: "Formatted at 2026-09-25 22:46:14 on dist-test-slave-5vdr"
server_key: "02f43aa0830ab9e1a0f06557231be321"
server_key_iv: "8fd08ebab212fda8549fa42e70350721"
server_key_version: "kuduclusterkey@0"
I20260925 22:46:14.248772 22482 fs_manager.cc:691] Time spent creating directory manager: real 0.008s user 0.006s sys 0.000s
I20260925 22:46:14.272449 22498 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:46:14.274530 22482 fs_manager.cc:725] Time spent opening block manager: real 0.006s user 0.002s sys 0.000s
I20260925 22:46:14.274716 22482 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376259650642-11032-0/minicluster-data/ts-1/data,/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376259650642-11032-0/minicluster-data/ts-1/wal
uuid: "176c4879d00447ecafb59a251a267394"
format_stamp: "Formatted at 2026-09-25 22:46:14 on dist-test-slave-5vdr"
server_key: "02f43aa0830ab9e1a0f06557231be321"
server_key_iv: "8fd08ebab212fda8549fa42e70350721"
server_key_version: "kuduclusterkey@0"
I20260925 22:46:14.274873 22482 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376259650642-11032-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376259650642-11032-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376259650642-11032-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:46:14.311277 22482 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:46:14.315392 22482 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:46:14.315636 22482 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:46:14.316421 22482 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 22:46:14.317572 22482 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:46:14.317641 22482 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:46:14.317696 22482 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:46:14.317732 22482 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 22:46:14.329538 22482 rpc_server.cc:307] RPC server started. Bound to: 127.10.198.2:44559
I20260925 22:46:14.330899 22482 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376259650642-11032-0/minicluster-data/ts-1/data/info.pb
I20260925 22:46:14.332463 22611 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.198.2:44559 every 8 connection(s)
I20260925 22:46:14.340950 11032 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrWXm8X/build/debug/bin/kudu as pid 22482
I20260925 22:46:14.341120 11032 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376259650642-11032-0/minicluster-data/ts-1/wal/instance
I20260925 22:46:14.341455 11032 external_mini_cluster.cc:1428] Setting key 28de108aa92093cb8ada4f7d0931c90b
Sep 25 22:46:14 dist-test-slave-5vdr krb5kdc[21786](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376374, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.198.2@KRBTEST.COM for kudu/127.10.198.62@KRBTEST.COM
I20260925 22:46:14.368613 22478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:46:14.335974 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.198.2:45009 (local address 127.10.198.62:40283)
0925 22:46:14.336126 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0925 22:46:14.336131 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:46:14.336149 (+ 18us) server_negotiation.cc:407] Connection header received
0925 22:46:14.337670 (+ 1521us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:46:14.337674 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:46:14.337740 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:46:14.337850 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:46:14.339621 (+ 1771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:46:14.340558 (+ 937us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:46:14.344479 (+ 3921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:46:14.344833 (+ 354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:46:14.344944 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:46:14.363913 (+ 18969us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:46:14.363925 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:46:14.363928 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:46:14.363963 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:46:14.365642 (+ 1679us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:46:14.366454 (+ 812us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:46:14.366458 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:46:14.366460 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:46:14.366563 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:46:14.367051 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:46:14.367056 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:46:14.367057 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:46:14.367310 (+ 253us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:46:14.367399 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0925 22:46:14.368226 (+ 827us) server_negotiation.cc:299] Negotiation successful
0925 22:46:14.368407 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260925 22:46:14.370688 22614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:46:14.334632 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.198.62:40283 (local address 127.10.198.2:45009)
0925 22:46:14.335406 (+ 774us) negotiation.cc:107] Waiting for socket to connect
0925 22:46:14.335488 (+ 82us) client_negotiation.cc:174] Beginning negotiation
0925 22:46:14.337301 (+ 1813us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:46:14.338148 (+ 847us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:46:14.338162 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:46:14.338579 (+ 417us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:46:14.339383 (+ 804us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:46:14.339406 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:46:14.342055 (+ 2649us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:46:14.342059 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:46:14.344291 (+ 2232us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:46:14.344310 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:46:14.345210 (+ 900us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:46:14.345214 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:46:14.351711 (+ 6497us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:46:14.354808 (+ 3097us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:46:14.354831 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:46:14.363529 (+ 8698us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:46:14.365815 (+ 2286us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:46:14.365823 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:46:14.365846 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:46:14.366304 (+ 458us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:46:14.366689 (+ 385us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:46:14.366691 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:46:14.366694 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:46:14.366916 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:46:14.367476 (+ 560us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:46:14.367484 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:46:14.367898 (+ 414us) client_negotiation.cc:769] Sending connection context
0925 22:46:14.369093 (+ 1195us) client_negotiation.cc:240] Negotiation successful
0925 22:46:14.369527 (+ 434us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":463,"thread_start_us":221,"threads_started":1}
I20260925 22:46:14.372715 22612 heartbeater.cc:347] Connected to a master server at 127.10.198.62:40283
I20260925 22:46:14.373020 22612 heartbeater.cc:464] Registering TS with master...
I20260925 22:46:14.373664 22612 heartbeater.cc:511] Master 127.10.198.62:40283 requested a full tablet report, sending...
I20260925 22:46:14.375399 22262 ts_manager.cc:194] Registered new tserver with Master: 176c4879d00447ecafb59a251a267394 (127.10.198.2:44559)
I20260925 22:46:14.376386 22262 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.198.2@KRBTEST.COM'} at 127.10.198.2:45009
WARNING: no policy specified for kudu/127.10.198.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.198.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.198.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376259650642-11032-0/krb5kdc/kudu_127.10.198.3.keytab.
Entry for principal kudu/127.10.198.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376259650642-11032-0/krb5kdc/kudu_127.10.198.3.keytab.
I20260925 22:46:14.412840 11032 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrWXm8X/build/debug/bin/kudu
/tmp/dist-test-taskrWXm8X/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376259650642-11032-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376259650642-11032-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376259650642-11032-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376259650642-11032-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.10.198.20:55509/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.198.3:0
--local_ip_for_outbound_sockets=127.10.198.3
--webserver_interface=127.10.198.3
--webserver_port=0
--tserver_master_addrs=127.10.198.62:40283
--keytab_file=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376259650642-11032-0/krb5kdc/kudu_127.10.198.3.keytab
--principal=kudu/127.10.198.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.198.20:44819
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376259650642-11032-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376259650642-11032-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376259650642-11032-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 25 22:46:14 dist-test-slave-5vdr krb5kdc[21786](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376374, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.198.62@KRBTEST.COM for HTTP/127.10.198.20@KRBTEST.COM
W20260925 22:46:14.609866 22617 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:46:14.610476 22617 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:46:14.610582 22617 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:46:14.617240 22617 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:46:14.617406 22617 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:46:14.617542 22617 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.198.3
I20260925 22:46:14.626314 22617 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.198.20:44819
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376259650642-11032-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376259650642-11032-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.10.198.20:55509/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376259650642-11032-0/krb5kdc/kudu_127.10.198.3.keytab
--principal=kudu/127.10.198.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.198.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376259650642-11032-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.198.3
--webserver_port=0
--tserver_master_addrs=127.10.198.62:40283
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22617
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.198.3
--log_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376259650642-11032-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:09:16 UTC on ccycloud.quasar-afmhyw.root.comops.site
build id 4673379
I20260925 22:46:14.628219 22617 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:46:14.629675 22617 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:46:14.643682 22623 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:46:14.644160 22617 server_base.cc:1034] running on GCE node
W20260925 22:46:14.644548 22622 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:46:14.652132 22625 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:46:14.652884 22617 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:46:14.653859 22617 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:46:14.661298 22617 hybrid_clock.cc:648] HybridClock initialized: now 1790376374661197 us; error 111 us; skew 500 ppm
Sep 25 22:46:14 dist-test-slave-5vdr krb5kdc[21786](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376374, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.198.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:46:14.665496 22617 init.cc:377] Logged in from keytab as kudu/127.10.198.3@KRBTEST.COM (short username kudu)
I20260925 22:46:14.666788 22617 webserver.cc:460] Webserver started at http://127.10.198.3:44567/ using document root <none> and password file <none>
I20260925 22:46:14.667596 22617 fs_manager.cc:359] Metadata directory not provided
I20260925 22:46:14.667694 22617 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:46:14.667980 22617 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 25 22:46:14 dist-test-slave-5vdr krb5kdc[21786](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376374, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.198.3@KRBTEST.COM for HTTP/127.10.198.20@KRBTEST.COM
I20260925 22:46:14.685851 22617 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376259650642-11032-0/minicluster-data/ts-2/data/instance:
uuid: "3399a87ac8a14444af02610a4e5b6a3b"
format_stamp: "Formatted at 2026-09-25 22:46:14 on dist-test-slave-5vdr"
server_key: "d4f5af402123ef7481d7b32929380322"
server_key_iv: "8bdc886c7fc9166bf17065301c131697"
server_key_version: "kuduclusterkey@0"
I20260925 22:46:14.686702 22617 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376259650642-11032-0/minicluster-data/ts-2/wal/instance:
uuid: "3399a87ac8a14444af02610a4e5b6a3b"
format_stamp: "Formatted at 2026-09-25 22:46:14 on dist-test-slave-5vdr"
server_key: "d4f5af402123ef7481d7b32929380322"
server_key_iv: "8bdc886c7fc9166bf17065301c131697"
server_key_version: "kuduclusterkey@0"
I20260925 22:46:14.692479 22617 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260925 22:46:14.710901 22632 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:46:14.712718 22617 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260925 22:46:14.712873 22617 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376259650642-11032-0/minicluster-data/ts-2/data,/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376259650642-11032-0/minicluster-data/ts-2/wal
uuid: "3399a87ac8a14444af02610a4e5b6a3b"
format_stamp: "Formatted at 2026-09-25 22:46:14 on dist-test-slave-5vdr"
server_key: "d4f5af402123ef7481d7b32929380322"
server_key_iv: "8bdc886c7fc9166bf17065301c131697"
server_key_version: "kuduclusterkey@0"
I20260925 22:46:14.713029 22617 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376259650642-11032-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376259650642-11032-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376259650642-11032-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:46:14.752671 22617 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:46:14.757023 22617 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:46:14.757244 22617 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:46:14.757921 22617 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 22:46:14.759035 22617 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:46:14.759100 22617 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:46:14.759161 22617 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:46:14.759189 22617 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 22:46:14.772046 22617 rpc_server.cc:307] RPC server started. Bound to: 127.10.198.3:44993
I20260925 22:46:14.773319 22617 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376259650642-11032-0/minicluster-data/ts-2/data/info.pb
I20260925 22:46:14.779677 11032 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrWXm8X/build/debug/bin/kudu as pid 22617
I20260925 22:46:14.779803 11032 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376259650642-11032-0/minicluster-data/ts-2/wal/instance
I20260925 22:46:14.780287 11032 external_mini_cluster.cc:1428] Setting key fedf856a0b09c55eabfd990303122908
I20260925 22:46:14.776607 22745 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.198.3:44993 every 8 connection(s)
Sep 25 22:46:14 dist-test-slave-5vdr krb5kdc[21786](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376374, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.198.3@KRBTEST.COM for kudu/127.10.198.62@KRBTEST.COM
I20260925 22:46:14.799279 22478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:46:14.780324 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.198.3:33873 (local address 127.10.198.62:40283)
0925 22:46:14.780487 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0925 22:46:14.780492 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:46:14.783131 (+ 2639us) server_negotiation.cc:407] Connection header received
0925 22:46:14.784451 (+ 1320us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:46:14.784455 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:46:14.784515 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:46:14.784567 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:46:14.786356 (+ 1789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:46:14.787219 (+ 863us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:46:14.789123 (+ 1904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:46:14.789510 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:46:14.789591 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:46:14.792896 (+ 3305us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:46:14.792900 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:46:14.792903 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:46:14.792934 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:46:14.796594 (+ 3660us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:46:14.797364 (+ 770us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:46:14.797367 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:46:14.797369 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:46:14.797460 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:46:14.797879 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:46:14.797882 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:46:14.797884 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:46:14.798136 (+ 252us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:46:14.798218 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0925 22:46:14.798953 (+ 735us) server_negotiation.cc:299] Negotiation successful
0925 22:46:14.799109 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260925 22:46:14.801116 22748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:46:14.779088 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.198.62:40283 (local address 127.10.198.3:33873)
0925 22:46:14.782960 (+ 3872us) negotiation.cc:107] Waiting for socket to connect
0925 22:46:14.783031 (+ 71us) client_negotiation.cc:174] Beginning negotiation
0925 22:46:14.784185 (+ 1154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:46:14.785039 (+ 854us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:46:14.785050 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:46:14.785433 (+ 383us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:46:14.786161 (+ 728us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:46:14.786179 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:46:14.787397 (+ 1218us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:46:14.787400 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:46:14.788938 (+ 1538us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:46:14.788952 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:46:14.789856 (+ 904us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:46:14.789859 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:46:14.790088 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:46:14.790853 (+ 765us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:46:14.790876 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:46:14.792640 (+ 1764us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:46:14.796780 (+ 4140us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:46:14.796789 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:46:14.796812 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:46:14.797237 (+ 425us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:46:14.797569 (+ 332us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:46:14.797572 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:46:14.797574 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:46:14.797770 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:46:14.798290 (+ 520us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:46:14.798298 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:46:14.798662 (+ 364us) client_negotiation.cc:769] Sending connection context
0925 22:46:14.799540 (+ 878us) client_negotiation.cc:240] Negotiation successful
0925 22:46:14.799970 (+ 430us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3523,"thread_start_us":166,"threads_started":1}
I20260925 22:46:14.802605 22746 heartbeater.cc:347] Connected to a master server at 127.10.198.62:40283
I20260925 22:46:14.802896 22746 heartbeater.cc:464] Registering TS with master...
I20260925 22:46:14.803547 22746 heartbeater.cc:511] Master 127.10.198.62:40283 requested a full tablet report, sending...
I20260925 22:46:14.804840 22262 ts_manager.cc:194] Registered new tserver with Master: 3399a87ac8a14444af02610a4e5b6a3b (127.10.198.3:44993)
I20260925 22:46:14.805614 22262 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.198.3@KRBTEST.COM'} at 127.10.198.3:33873
I20260925 22:46:14.809813 11032 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 22:46:14.822821 22478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:46:14.812289 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44824 (local address 127.10.198.62:40283)
0925 22:46:14.812412 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0925 22:46:14.812417 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:46:14.813082 (+ 665us) server_negotiation.cc:407] Connection header received
0925 22:46:14.813286 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:46:14.813290 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:46:14.813342 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:46:14.813395 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:46:14.814439 (+ 1044us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:46:14.815255 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:46:14.816492 (+ 1237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:46:14.816810 (+ 318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:46:14.816887 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:46:14.818842 (+ 1955us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:46:14.818847 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:46:14.818850 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:46:14.818884 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:46:14.820749 (+ 1865us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:46:14.821373 (+ 624us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:46:14.821377 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:46:14.821379 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:46:14.821482 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:46:14.821799 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:46:14.821802 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:46:14.821804 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:46:14.822036 (+ 232us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:46:14.822157 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0925 22:46:14.822522 (+ 365us) server_negotiation.cc:299] Negotiation successful
0925 22:46:14.822659 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260925 22:46:14.827013 22262 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44824:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 22:46:14.831627 22262 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 22:46:14.865437 22761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:46:14.853316 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.198.3:44993 (local address 127.0.0.1:50456)
0925 22:46:14.855690 (+ 2374us) negotiation.cc:107] Waiting for socket to connect
0925 22:46:14.855702 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 22:46:14.855762 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:46:14.857760 (+ 1998us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:46:14.857768 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:46:14.857794 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:46:14.857962 (+ 168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:46:14.857968 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:46:14.859618 (+ 1650us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:46:14.859622 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:46:14.862227 (+ 2605us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:46:14.862243 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:46:14.863583 (+ 1340us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:46:14.863588 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:46:14.863735 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:46:14.863787 (+ 52us) client_negotiation.cc:769] Sending connection context
0925 22:46:14.863929 (+ 142us) client_negotiation.cc:240] Negotiation successful
0925 22:46:14.865257 (+ 1328us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2327,"thread_start_us":97,"threads_started":1}
I20260925 22:46:14.866905 22764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:46:14.853486 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50456 (local address 127.10.198.3:44993)
0925 22:46:14.856121 (+ 2635us) server_negotiation.cc:206] Beginning negotiation
0925 22:46:14.856138 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0925 22:46:14.856169 (+ 31us) server_negotiation.cc:407] Connection header received
0925 22:46:14.856256 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:46:14.856263 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:46:14.857548 (+ 1285us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:46:14.858161 (+ 613us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:46:14.858311 (+ 150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:46:14.859455 (+ 1144us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:46:14.862464 (+ 3009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:46:14.863402 (+ 938us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:46:14.866368 (+ 2966us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:46:14.866565 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0925 22:46:14.866629 (+ 64us) server_negotiation.cc:299] Negotiation successful
0925 22:46:14.866745 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2495,"thread_start_us":98,"threads_started":1}
I20260925 22:46:14.870605 22680 tablet_service.cc:1467] Processing CreateTablet for tablet abcee66fa0ea42b38ac9a1d879b982ca (DEFAULT_TABLE table=test-table [id=7136ee13911c45efb91f87dff4fa00a0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 22:46:14.873276 22680 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet abcee66fa0ea42b38ac9a1d879b982ca. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:46:14.874064 22763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:46:14.850126 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.198.1:36929 (local address 127.0.0.1:47338)
0925 22:46:14.854965 (+ 4839us) negotiation.cc:107] Waiting for socket to connect
0925 22:46:14.855045 (+ 80us) client_negotiation.cc:174] Beginning negotiation
0925 22:46:14.855222 (+ 177us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:46:14.865133 (+ 9911us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:46:14.865136 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:46:14.865159 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:46:14.865851 (+ 692us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:46:14.865857 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:46:14.868527 (+ 2670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:46:14.868531 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:46:14.869685 (+ 1154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:46:14.869702 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:46:14.873599 (+ 3897us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:46:14.873604 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:46:14.873708 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:46:14.873740 (+ 32us) client_negotiation.cc:769] Sending connection context
0925 22:46:14.873799 (+ 59us) client_negotiation.cc:240] Negotiation successful
0925 22:46:14.873890 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4618,"mutex_wait_us":593,"thread_start_us":140,"threads_started":1}
I20260925 22:46:14.874972 22759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:46:14.850199 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47338 (local address 127.10.198.1:36929)
0925 22:46:14.864103 (+ 13904us) server_negotiation.cc:206] Beginning negotiation
0925 22:46:14.864111 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 22:46:14.864140 (+ 29us) server_negotiation.cc:407] Connection header received
0925 22:46:14.864228 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:46:14.864237 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:46:14.864506 (+ 269us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:46:14.864696 (+ 190us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:46:14.867229 (+ 2533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:46:14.868372 (+ 1143us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:46:14.869987 (+ 1615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:46:14.870837 (+ 850us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:46:14.874406 (+ 3569us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:46:14.874634 (+ 228us) server_negotiation.cc:1035] Waiting for connection context
0925 22:46:14.874704 (+ 70us) server_negotiation.cc:299] Negotiation successful
0925 22:46:14.874822 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13761,"thread_start_us":87,"threads_started":1}
I20260925 22:46:14.875682 22760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:46:14.852677 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.198.2:44559 (local address 127.0.0.1:35304)
0925 22:46:14.855457 (+ 2780us) negotiation.cc:107] Waiting for socket to connect
0925 22:46:14.855471 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 22:46:14.855540 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:46:14.865524 (+ 9984us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:46:14.865527 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:46:14.865548 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:46:14.865688 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:46:14.865694 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:46:14.871032 (+ 5338us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:46:14.871036 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:46:14.872119 (+ 1083us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:46:14.872129 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:46:14.875237 (+ 3108us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:46:14.875242 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:46:14.875359 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:46:14.875387 (+ 28us) client_negotiation.cc:769] Sending connection context
0925 22:46:14.875446 (+ 59us) client_negotiation.cc:240] Negotiation successful
0925 22:46:14.875530 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2717,"thread_start_us":124,"threads_started":1}
I20260925 22:46:14.877568 22762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:46:14.852849 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35304 (local address 127.10.198.2:44559)
0925 22:46:14.865027 (+ 12178us) server_negotiation.cc:206] Beginning negotiation
0925 22:46:14.865047 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0925 22:46:14.865074 (+ 27us) server_negotiation.cc:407] Connection header received
0925 22:46:14.865150 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:46:14.865155 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:46:14.865357 (+ 202us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:46:14.865846 (+ 489us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:46:14.866004 (+ 158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:46:14.867073 (+ 1069us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:46:14.872545 (+ 5472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:46:14.873392 (+ 847us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:46:14.876025 (+ 2633us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:46:14.876236 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0925 22:46:14.876297 (+ 61us) server_negotiation.cc:299] Negotiation successful
0925 22:46:14.876404 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12054,"thread_start_us":92,"threads_started":1}
I20260925 22:46:14.879999 22410 tablet_service.cc:1467] Processing CreateTablet for tablet abcee66fa0ea42b38ac9a1d879b982ca (DEFAULT_TABLE table=test-table [id=7136ee13911c45efb91f87dff4fa00a0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 22:46:14.880605 22546 tablet_service.cc:1467] Processing CreateTablet for tablet abcee66fa0ea42b38ac9a1d879b982ca (DEFAULT_TABLE table=test-table [id=7136ee13911c45efb91f87dff4fa00a0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 22:46:14.881493 22410 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet abcee66fa0ea42b38ac9a1d879b982ca. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:46:14.881731 22546 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet abcee66fa0ea42b38ac9a1d879b982ca. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:46:14.887218 22765 tablet_bootstrap.cc:492] T abcee66fa0ea42b38ac9a1d879b982ca P 3399a87ac8a14444af02610a4e5b6a3b: Bootstrap starting.
I20260925 22:46:14.890548 22765 tablet_bootstrap.cc:654] T abcee66fa0ea42b38ac9a1d879b982ca P 3399a87ac8a14444af02610a4e5b6a3b: Neither blocks nor log segments found. Creating new log.
I20260925 22:46:14.891564 22765 log.cc:824] T abcee66fa0ea42b38ac9a1d879b982ca P 3399a87ac8a14444af02610a4e5b6a3b: Log is configured to *not* fsync() on all Append() calls
I20260925 22:46:14.894050 22765 tablet_bootstrap.cc:492] T abcee66fa0ea42b38ac9a1d879b982ca P 3399a87ac8a14444af02610a4e5b6a3b: No bootstrap required, opened a new log
I20260925 22:46:14.894311 22765 ts_tablet_manager.cc:1397] T abcee66fa0ea42b38ac9a1d879b982ca P 3399a87ac8a14444af02610a4e5b6a3b: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20260925 22:46:14.899780 22765 raft_consensus.cc:348] T abcee66fa0ea42b38ac9a1d879b982ca P 3399a87ac8a14444af02610a4e5b6a3b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6972169c9517492eaf91bc5ec8e851b7" member_type: VOTER last_known_addr { host: "127.10.198.1" port: 36929 } } peers { permanent_uuid: "176c4879d00447ecafb59a251a267394" member_type: VOTER last_known_addr { host: "127.10.198.2" port: 44559 } } peers { permanent_uuid: "3399a87ac8a14444af02610a4e5b6a3b" member_type: VOTER last_known_addr { host: "127.10.198.3" port: 44993 } }
I20260925 22:46:14.900254 22765 raft_consensus.cc:374] T abcee66fa0ea42b38ac9a1d879b982ca P 3399a87ac8a14444af02610a4e5b6a3b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:46:14.900315 22765 raft_consensus.cc:729] T abcee66fa0ea42b38ac9a1d879b982ca P 3399a87ac8a14444af02610a4e5b6a3b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3399a87ac8a14444af02610a4e5b6a3b, State: Initialized, Role: FOLLOWER
I20260925 22:46:14.900884 22765 consensus_queue.cc:260] T abcee66fa0ea42b38ac9a1d879b982ca P 3399a87ac8a14444af02610a4e5b6a3b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6972169c9517492eaf91bc5ec8e851b7" member_type: VOTER last_known_addr { host: "127.10.198.1" port: 36929 } } peers { permanent_uuid: "176c4879d00447ecafb59a251a267394" member_type: VOTER last_known_addr { host: "127.10.198.2" port: 44559 } } peers { permanent_uuid: "3399a87ac8a14444af02610a4e5b6a3b" member_type: VOTER last_known_addr { host: "127.10.198.3" port: 44993 } }
I20260925 22:46:14.901751 22766 tablet_bootstrap.cc:492] T abcee66fa0ea42b38ac9a1d879b982ca P 176c4879d00447ecafb59a251a267394: Bootstrap starting.
I20260925 22:46:14.902194 22765 ts_tablet_manager.cc:1428] T abcee66fa0ea42b38ac9a1d879b982ca P 3399a87ac8a14444af02610a4e5b6a3b: Time spent starting tablet: real 0.008s user 0.006s sys 0.000s
I20260925 22:46:14.902586 22746 heartbeater.cc:503] Master 127.10.198.62:40283 was elected leader, sending a full tablet report...
I20260925 22:46:14.904412 22766 tablet_bootstrap.cc:654] T abcee66fa0ea42b38ac9a1d879b982ca P 176c4879d00447ecafb59a251a267394: Neither blocks nor log segments found. Creating new log.
I20260925 22:46:14.905385 22766 log.cc:824] T abcee66fa0ea42b38ac9a1d879b982ca P 176c4879d00447ecafb59a251a267394: Log is configured to *not* fsync() on all Append() calls
I20260925 22:46:14.910094 22768 tablet_bootstrap.cc:492] T abcee66fa0ea42b38ac9a1d879b982ca P 6972169c9517492eaf91bc5ec8e851b7: Bootstrap starting.
I20260925 22:46:14.913048 22768 tablet_bootstrap.cc:654] T abcee66fa0ea42b38ac9a1d879b982ca P 6972169c9517492eaf91bc5ec8e851b7: Neither blocks nor log segments found. Creating new log.
I20260925 22:46:14.914044 22768 log.cc:824] T abcee66fa0ea42b38ac9a1d879b982ca P 6972169c9517492eaf91bc5ec8e851b7: Log is configured to *not* fsync() on all Append() calls
I20260925 22:46:14.916659 22768 tablet_bootstrap.cc:492] T abcee66fa0ea42b38ac9a1d879b982ca P 6972169c9517492eaf91bc5ec8e851b7: No bootstrap required, opened a new log
I20260925 22:46:14.916904 22768 ts_tablet_manager.cc:1397] T abcee66fa0ea42b38ac9a1d879b982ca P 6972169c9517492eaf91bc5ec8e851b7: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20260925 22:46:14.917793 22766 tablet_bootstrap.cc:492] T abcee66fa0ea42b38ac9a1d879b982ca P 176c4879d00447ecafb59a251a267394: No bootstrap required, opened a new log
I20260925 22:46:14.918020 22766 ts_tablet_manager.cc:1397] T abcee66fa0ea42b38ac9a1d879b982ca P 176c4879d00447ecafb59a251a267394: Time spent bootstrapping tablet: real 0.016s user 0.004s sys 0.000s
I20260925 22:46:14.923063 22766 raft_consensus.cc:348] T abcee66fa0ea42b38ac9a1d879b982ca P 176c4879d00447ecafb59a251a267394 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6972169c9517492eaf91bc5ec8e851b7" member_type: VOTER last_known_addr { host: "127.10.198.1" port: 36929 } } peers { permanent_uuid: "176c4879d00447ecafb59a251a267394" member_type: VOTER last_known_addr { host: "127.10.198.2" port: 44559 } } peers { permanent_uuid: "3399a87ac8a14444af02610a4e5b6a3b" member_type: VOTER last_known_addr { host: "127.10.198.3" port: 44993 } }
I20260925 22:46:14.923213 22768 raft_consensus.cc:348] T abcee66fa0ea42b38ac9a1d879b982ca P 6972169c9517492eaf91bc5ec8e851b7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6972169c9517492eaf91bc5ec8e851b7" member_type: VOTER last_known_addr { host: "127.10.198.1" port: 36929 } } peers { permanent_uuid: "176c4879d00447ecafb59a251a267394" member_type: VOTER last_known_addr { host: "127.10.198.2" port: 44559 } } peers { permanent_uuid: "3399a87ac8a14444af02610a4e5b6a3b" member_type: VOTER last_known_addr { host: "127.10.198.3" port: 44993 } }
I20260925 22:46:14.923455 22768 raft_consensus.cc:374] T abcee66fa0ea42b38ac9a1d879b982ca P 6972169c9517492eaf91bc5ec8e851b7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:46:14.923455 22766 raft_consensus.cc:374] T abcee66fa0ea42b38ac9a1d879b982ca P 176c4879d00447ecafb59a251a267394 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:46:14.923514 22768 raft_consensus.cc:729] T abcee66fa0ea42b38ac9a1d879b982ca P 6972169c9517492eaf91bc5ec8e851b7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6972169c9517492eaf91bc5ec8e851b7, State: Initialized, Role: FOLLOWER
I20260925 22:46:14.923514 22766 raft_consensus.cc:729] T abcee66fa0ea42b38ac9a1d879b982ca P 176c4879d00447ecafb59a251a267394 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 176c4879d00447ecafb59a251a267394, State: Initialized, Role: FOLLOWER
I20260925 22:46:14.924129 22766 consensus_queue.cc:260] T abcee66fa0ea42b38ac9a1d879b982ca P 176c4879d00447ecafb59a251a267394 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6972169c9517492eaf91bc5ec8e851b7" member_type: VOTER last_known_addr { host: "127.10.198.1" port: 36929 } } peers { permanent_uuid: "176c4879d00447ecafb59a251a267394" member_type: VOTER last_known_addr { host: "127.10.198.2" port: 44559 } } peers { permanent_uuid: "3399a87ac8a14444af02610a4e5b6a3b" member_type: VOTER last_known_addr { host: "127.10.198.3" port: 44993 } }
I20260925 22:46:14.924129 22768 consensus_queue.cc:260] T abcee66fa0ea42b38ac9a1d879b982ca P 6972169c9517492eaf91bc5ec8e851b7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6972169c9517492eaf91bc5ec8e851b7" member_type: VOTER last_known_addr { host: "127.10.198.1" port: 36929 } } peers { permanent_uuid: "176c4879d00447ecafb59a251a267394" member_type: VOTER last_known_addr { host: "127.10.198.2" port: 44559 } } peers { permanent_uuid: "3399a87ac8a14444af02610a4e5b6a3b" member_type: VOTER last_known_addr { host: "127.10.198.3" port: 44993 } }
I20260925 22:46:14.925379 22768 ts_tablet_manager.cc:1428] T abcee66fa0ea42b38ac9a1d879b982ca P 6972169c9517492eaf91bc5ec8e851b7: Time spent starting tablet: real 0.008s user 0.007s sys 0.000s
I20260925 22:46:14.925379 22766 ts_tablet_manager.cc:1428] T abcee66fa0ea42b38ac9a1d879b982ca P 176c4879d00447ecafb59a251a267394: Time spent starting tablet: real 0.007s user 0.008s sys 0.000s
I20260925 22:46:14.925829 22476 heartbeater.cc:503] Master 127.10.198.62:40283 was elected leader, sending a full tablet report...
I20260925 22:46:14.925990 22612 heartbeater.cc:503] Master 127.10.198.62:40283 was elected leader, sending a full tablet report...
W20260925 22:46:14.940063 22477 tablet.cc:2369] T abcee66fa0ea42b38ac9a1d879b982ca P 6972169c9517492eaf91bc5ec8e851b7: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 25 22:46:14 dist-test-slave-5vdr krb5kdc[21786](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376374, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.198.62@KRBTEST.COM for HTTP/127.10.198.20@KRBTEST.COM
W20260925 22:46:15.032835 22747 tablet.cc:2369] T abcee66fa0ea42b38ac9a1d879b982ca P 3399a87ac8a14444af02610a4e5b6a3b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 22:46:15.065586 22772 raft_consensus.cc:482] T abcee66fa0ea42b38ac9a1d879b982ca P 6972169c9517492eaf91bc5ec8e851b7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 22:46:15.065840 22772 raft_consensus.cc:504] T abcee66fa0ea42b38ac9a1d879b982ca P 6972169c9517492eaf91bc5ec8e851b7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6972169c9517492eaf91bc5ec8e851b7" member_type: VOTER last_known_addr { host: "127.10.198.1" port: 36929 } } peers { permanent_uuid: "176c4879d00447ecafb59a251a267394" member_type: VOTER last_known_addr { host: "127.10.198.2" port: 44559 } } peers { permanent_uuid: "3399a87ac8a14444af02610a4e5b6a3b" member_type: VOTER last_known_addr { host: "127.10.198.3" port: 44993 } }
I20260925 22:46:15.067301 22772 leader_election.cc:288] T abcee66fa0ea42b38ac9a1d879b982ca P 6972169c9517492eaf91bc5ec8e851b7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 176c4879d00447ecafb59a251a267394 (127.10.198.2:44559), 3399a87ac8a14444af02610a4e5b6a3b (127.10.198.3:44993)
I20260925 22:46:15.079221 22762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:46:15.068072 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.198.1:48815 (local address 127.10.198.2:44559)
0925 22:46:15.068198 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0925 22:46:15.068203 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:46:15.068219 (+ 16us) server_negotiation.cc:407] Connection header received
0925 22:46:15.068500 (+ 281us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:46:15.068504 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:46:15.068560 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:46:15.068611 (+ 51us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:46:15.069039 (+ 428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:46:15.069825 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:46:15.074532 (+ 4707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:46:15.075278 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:46:15.075412 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:46:15.075471 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0925 22:46:15.078933 (+ 3462us) server_negotiation.cc:299] Negotiation successful
0925 22:46:15.079053 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260925 22:46:15.080153 22777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:46:15.067566 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.198.2:44559 (local address 127.10.198.1:48815)
0925 22:46:15.068054 (+ 488us) negotiation.cc:107] Waiting for socket to connect
0925 22:46:15.068070 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 22:46:15.068331 (+ 261us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:46:15.068773 (+ 442us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:46:15.068776 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:46:15.068796 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:46:15.068905 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:46:15.068911 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:46:15.070940 (+ 2029us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:46:15.070944 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:46:15.073941 (+ 2997us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:46:15.073964 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:46:15.078611 (+ 4647us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:46:15.078614 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:46:15.078732 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:46:15.078774 (+ 42us) client_negotiation.cc:769] Sending connection context
0925 22:46:15.079810 (+ 1036us) client_negotiation.cc:240] Negotiation successful
0925 22:46:15.079952 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":407,"thread_start_us":139,"threads_started":1}
I20260925 22:46:15.081212 22566 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "abcee66fa0ea42b38ac9a1d879b982ca" candidate_uuid: "6972169c9517492eaf91bc5ec8e851b7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "176c4879d00447ecafb59a251a267394" is_pre_election: true
I20260925 22:46:15.081588 22566 raft_consensus.cc:2454] T abcee66fa0ea42b38ac9a1d879b982ca P 176c4879d00447ecafb59a251a267394 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6972169c9517492eaf91bc5ec8e851b7 in term 0.
I20260925 22:46:15.082443 22364 leader_election.cc:302] T abcee66fa0ea42b38ac9a1d879b982ca P 6972169c9517492eaf91bc5ec8e851b7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 176c4879d00447ecafb59a251a267394, 6972169c9517492eaf91bc5ec8e851b7; no voters:
I20260925 22:46:15.082834 22772 raft_consensus.cc:2781] T abcee66fa0ea42b38ac9a1d879b982ca P 6972169c9517492eaf91bc5ec8e851b7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 22:46:15.082923 22772 raft_consensus.cc:482] T abcee66fa0ea42b38ac9a1d879b982ca P 6972169c9517492eaf91bc5ec8e851b7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 22:46:15.082968 22772 raft_consensus.cc:3037] T abcee66fa0ea42b38ac9a1d879b982ca P 6972169c9517492eaf91bc5ec8e851b7 [term 0 FOLLOWER]: Advancing to term 1
W20260925 22:46:15.083557 22613 tablet.cc:2369] T abcee66fa0ea42b38ac9a1d879b982ca P 176c4879d00447ecafb59a251a267394: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 22:46:15.084662 22772 raft_consensus.cc:504] T abcee66fa0ea42b38ac9a1d879b982ca P 6972169c9517492eaf91bc5ec8e851b7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6972169c9517492eaf91bc5ec8e851b7" member_type: VOTER last_known_addr { host: "127.10.198.1" port: 36929 } } peers { permanent_uuid: "176c4879d00447ecafb59a251a267394" member_type: VOTER last_known_addr { host: "127.10.198.2" port: 44559 } } peers { permanent_uuid: "3399a87ac8a14444af02610a4e5b6a3b" member_type: VOTER last_known_addr { host: "127.10.198.3" port: 44993 } }
I20260925 22:46:15.085153 22772 leader_election.cc:288] T abcee66fa0ea42b38ac9a1d879b982ca P 6972169c9517492eaf91bc5ec8e851b7 [CANDIDATE]: Term 1 election: Requested vote from peers 176c4879d00447ecafb59a251a267394 (127.10.198.2:44559), 3399a87ac8a14444af02610a4e5b6a3b (127.10.198.3:44993)
I20260925 22:46:15.085904 22566 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "abcee66fa0ea42b38ac9a1d879b982ca" candidate_uuid: "6972169c9517492eaf91bc5ec8e851b7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "176c4879d00447ecafb59a251a267394"
I20260925 22:46:15.086057 22566 raft_consensus.cc:3037] T abcee66fa0ea42b38ac9a1d879b982ca P 176c4879d00447ecafb59a251a267394 [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:46:15.087613 22566 raft_consensus.cc:2454] T abcee66fa0ea42b38ac9a1d879b982ca P 176c4879d00447ecafb59a251a267394 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6972169c9517492eaf91bc5ec8e851b7 in term 1.
I20260925 22:46:15.088295 22364 leader_election.cc:302] T abcee66fa0ea42b38ac9a1d879b982ca P 6972169c9517492eaf91bc5ec8e851b7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 176c4879d00447ecafb59a251a267394, 6972169c9517492eaf91bc5ec8e851b7; no voters:
I20260925 22:46:15.088614 22772 raft_consensus.cc:2781] T abcee66fa0ea42b38ac9a1d879b982ca P 6972169c9517492eaf91bc5ec8e851b7 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 22:46:15.088855 22772 raft_consensus.cc:686] T abcee66fa0ea42b38ac9a1d879b982ca P 6972169c9517492eaf91bc5ec8e851b7 [term 1 LEADER]: Becoming Leader. State: Replica: 6972169c9517492eaf91bc5ec8e851b7, State: Running, Role: LEADER
I20260925 22:46:15.089285 22772 consensus_queue.cc:237] T abcee66fa0ea42b38ac9a1d879b982ca P 6972169c9517492eaf91bc5ec8e851b7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6972169c9517492eaf91bc5ec8e851b7" member_type: VOTER last_known_addr { host: "127.10.198.1" port: 36929 } } peers { permanent_uuid: "176c4879d00447ecafb59a251a267394" member_type: VOTER last_known_addr { host: "127.10.198.2" port: 44559 } } peers { permanent_uuid: "3399a87ac8a14444af02610a4e5b6a3b" member_type: VOTER last_known_addr { host: "127.10.198.3" port: 44993 } }
I20260925 22:46:15.093536 22261 catalog_manager.cc:5495] T abcee66fa0ea42b38ac9a1d879b982ca P 6972169c9517492eaf91bc5ec8e851b7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6972169c9517492eaf91bc5ec8e851b7 (127.10.198.1). New cstate: current_term: 1 leader_uuid: "6972169c9517492eaf91bc5ec8e851b7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6972169c9517492eaf91bc5ec8e851b7" member_type: VOTER last_known_addr { host: "127.10.198.1" port: 36929 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "176c4879d00447ecafb59a251a267394" member_type: VOTER last_known_addr { host: "127.10.198.2" port: 44559 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3399a87ac8a14444af02610a4e5b6a3b" member_type: VOTER last_known_addr { host: "127.10.198.3" port: 44993 } health_report { overall_health: UNKNOWN } } }
I20260925 22:46:15.099879 22764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:46:15.068127 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.198.1:58337 (local address 127.10.198.3:44993)
0925 22:46:15.068266 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0925 22:46:15.068270 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:46:15.074142 (+ 5872us) server_negotiation.cc:407] Connection header received
0925 22:46:15.074269 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:46:15.074272 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:46:15.074331 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:46:15.074406 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:46:15.075944 (+ 1538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:46:15.076811 (+ 867us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:46:15.081056 (+ 4245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:46:15.081713 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:46:15.081830 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:46:15.081888 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0925 22:46:15.099581 (+ 17693us) server_negotiation.cc:299] Negotiation successful
0925 22:46:15.099697 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260925 22:46:15.100708 22776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:46:15.067682 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.198.3:44993 (local address 127.10.198.1:58337)
0925 22:46:15.074014 (+ 6332us) negotiation.cc:107] Waiting for socket to connect
0925 22:46:15.074035 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0925 22:46:15.074153 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:46:15.075553 (+ 1400us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:46:15.075556 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:46:15.075589 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:46:15.075732 (+ 143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:46:15.075741 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:46:15.077028 (+ 1287us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:46:15.077032 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:46:15.078357 (+ 1325us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:46:15.078374 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:46:15.099220 (+ 20846us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:46:15.099224 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:46:15.099376 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:46:15.099409 (+ 33us) client_negotiation.cc:769] Sending connection context
0925 22:46:15.100409 (+ 1000us) client_negotiation.cc:240] Negotiation successful
0925 22:46:15.100528 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6228,"thread_start_us":107,"threads_started":1}
I20260925 22:46:15.101770 22685 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "abcee66fa0ea42b38ac9a1d879b982ca" candidate_uuid: "6972169c9517492eaf91bc5ec8e851b7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3399a87ac8a14444af02610a4e5b6a3b" is_pre_election: true
I20260925 22:46:15.102156 22685 raft_consensus.cc:2454] T abcee66fa0ea42b38ac9a1d879b982ca P 3399a87ac8a14444af02610a4e5b6a3b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6972169c9517492eaf91bc5ec8e851b7 in term 0.
I20260925 22:46:15.102545 22698 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "abcee66fa0ea42b38ac9a1d879b982ca" candidate_uuid: "6972169c9517492eaf91bc5ec8e851b7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3399a87ac8a14444af02610a4e5b6a3b"
I20260925 22:46:15.102690 22698 raft_consensus.cc:3037] T abcee66fa0ea42b38ac9a1d879b982ca P 3399a87ac8a14444af02610a4e5b6a3b [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:46:15.104163 22698 raft_consensus.cc:2454] T abcee66fa0ea42b38ac9a1d879b982ca P 3399a87ac8a14444af02610a4e5b6a3b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6972169c9517492eaf91bc5ec8e851b7 in term 1.
I20260925 22:46:15.166481 22759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:46:15.160759 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47346 (local address 127.10.198.1:36929)
0925 22:46:15.160884 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0925 22:46:15.160902 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0925 22:46:15.161169 (+ 267us) server_negotiation.cc:407] Connection header received
0925 22:46:15.161279 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:46:15.161282 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:46:15.161356 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:46:15.161461 (+ 105us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 22:46:15.161890 (+ 429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:46:15.162714 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:46:15.164119 (+ 1405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:46:15.164458 (+ 339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:46:15.164579 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:46:15.165063 (+ 484us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 22:46:15.165681 (+ 618us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 22:46:15.165823 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0925 22:46:15.166225 (+ 402us) server_negotiation.cc:299] Negotiation successful
0925 22:46:15.166304 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260925 22:46:15.175416 22698 raft_consensus.cc:1261] T abcee66fa0ea42b38ac9a1d879b982ca P 3399a87ac8a14444af02610a4e5b6a3b [term 1 FOLLOWER]: Refusing update from remote peer 6972169c9517492eaf91bc5ec8e851b7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 22:46:15.176838 22778 consensus_queue.cc:1035] T abcee66fa0ea42b38ac9a1d879b982ca P 6972169c9517492eaf91bc5ec8e851b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3399a87ac8a14444af02610a4e5b6a3b" member_type: VOTER last_known_addr { host: "127.10.198.3" port: 44993 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 22:46:15.186560 22566 raft_consensus.cc:1261] T abcee66fa0ea42b38ac9a1d879b982ca P 176c4879d00447ecafb59a251a267394 [term 1 FOLLOWER]: Refusing update from remote peer 6972169c9517492eaf91bc5ec8e851b7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 22:46:15.187599 22778 consensus_queue.cc:1035] T abcee66fa0ea42b38ac9a1d879b982ca P 6972169c9517492eaf91bc5ec8e851b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "176c4879d00447ecafb59a251a267394" member_type: VOTER last_known_addr { host: "127.10.198.2" port: 44559 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 22:46:15.203136 22782 mvcc.cc:205] Tried to move back new op lower bound from 7333381632707334144 to 7333381632369844224. Current Snapshot: MvccSnapshot[applied={T|T < 7333381632707334144}]
I20260925 22:46:15.204932 22781 mvcc.cc:205] Tried to move back new op lower bound from 7333381632707334144 to 7333381632369844224. Current Snapshot: MvccSnapshot[applied={T|T < 7333381632707334144}]
I20260925 22:46:15.219586 22784 mvcc.cc:205] Tried to move back new op lower bound from 7333381632707334144 to 7333381632369844224. Current Snapshot: MvccSnapshot[applied={T|T < 7333381632707334144 or (T in {7333381632707334144})}]
I20260925 22:46:15.229775 22261 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44824:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 22:46:15.229983 22261 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44824:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 22:46:15.233505 22261 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 68f68a9d8f944d0fbc86369d1685d646: Sending DeleteTablet for 3 replicas of tablet abcee66fa0ea42b38ac9a1d879b982ca
I20260925 22:46:15.234771 22680 tablet_service.cc:1514] Processing DeleteTablet for tablet abcee66fa0ea42b38ac9a1d879b982ca with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 22:46:15 UTC) from {username='kudu', principal='kudu/127.10.198.62@KRBTEST.COM'} at 127.0.0.1:50456
I20260925 22:46:15.234797 22410 tablet_service.cc:1514] Processing DeleteTablet for tablet abcee66fa0ea42b38ac9a1d879b982ca with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 22:46:15 UTC) from {username='kudu', principal='kudu/127.10.198.62@KRBTEST.COM'} at 127.0.0.1:47338
I20260925 22:46:15.235801 22546 tablet_service.cc:1514] Processing DeleteTablet for tablet abcee66fa0ea42b38ac9a1d879b982ca with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 22:46:15 UTC) from {username='kudu', principal='kudu/127.10.198.62@KRBTEST.COM'} at 127.0.0.1:35304
I20260925 22:46:15.236979 11032 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrWXm8X/build/debug/bin/kudu with pid 22343
I20260925 22:46:15.241688 22791 tablet_replica.cc:330] T abcee66fa0ea42b38ac9a1d879b982ca P 3399a87ac8a14444af02610a4e5b6a3b: stopping tablet replica
I20260925 22:46:15.242228 22791 raft_consensus.cc:2229] T abcee66fa0ea42b38ac9a1d879b982ca P 3399a87ac8a14444af02610a4e5b6a3b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 22:46:15.242750 22791 raft_consensus.cc:2258] T abcee66fa0ea42b38ac9a1d879b982ca P 3399a87ac8a14444af02610a4e5b6a3b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 22:46:15.244131 22792 tablet_replica.cc:330] T abcee66fa0ea42b38ac9a1d879b982ca P 176c4879d00447ecafb59a251a267394: stopping tablet replica
I20260925 22:46:15.244504 22792 raft_consensus.cc:2229] T abcee66fa0ea42b38ac9a1d879b982ca P 176c4879d00447ecafb59a251a267394 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 22:46:15.244905 22792 raft_consensus.cc:2258] T abcee66fa0ea42b38ac9a1d879b982ca P 176c4879d00447ecafb59a251a267394 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 22:46:15.245379 22791 ts_tablet_manager.cc:1905] T abcee66fa0ea42b38ac9a1d879b982ca P 3399a87ac8a14444af02610a4e5b6a3b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 22:46:15.254689 22791 ts_tablet_manager.cc:1918] T abcee66fa0ea42b38ac9a1d879b982ca P 3399a87ac8a14444af02610a4e5b6a3b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 22:46:15.254987 22791 log.cc:1196] T abcee66fa0ea42b38ac9a1d879b982ca P 3399a87ac8a14444af02610a4e5b6a3b: Deleting WAL directory at /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790376259650642-11032-0/minicluster-data/ts-2/wal/wals/abcee66fa0ea42b38ac9a1d879b982ca
I20260925 22:46:15.255506 22791 ts_tablet_manager.cc:1939] T abcee66fa0ea42b38ac9a1d879b982ca P 3399a87ac8a14444af02610a4e5b6a3b: Deleting consensus metadata
W20260925 22:46:15.256714 22248 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.10.198.1:36929 (error 108)
I20260925 22:46:15.256920 22792 ts_tablet_manager.cc:1905] T abcee66fa0ea42b38ac9a1d879b982ca P 176c4879d00447ecafb59a251a267394: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 22:46:15.258903 22247 catalog_manager.cc:4841] TS 3399a87ac8a14444af02610a4e5b6a3b (127.10.198.3:44993): tablet abcee66fa0ea42b38ac9a1d879b982ca (table test-table [id=7136ee13911c45efb91f87dff4fa00a0]) successfully deleted
I20260925 22:46:15.259727 11032 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrWXm8X/build/debug/bin/kudu with pid 22482
I20260925 22:46:15.261806 22760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:46:15.261354 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.198.1:36929 (local address 127.0.0.1:47360)
0925 22:46:15.261589 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0925 22:46:15.261686 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.198.1:36929: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":131}
W20260925 22:46:15.262156 22248 catalog_manager.cc:4584] TS 6972169c9517492eaf91bc5ec8e851b7 (127.10.198.1:36929): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet abcee66fa0ea42b38ac9a1d879b982ca: Network error: Client connection negotiation failed: client connection to 127.10.198.1:36929: connect: Connection refused (error 111)
I20260925 22:46:15.267117 11032 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrWXm8X/build/debug/bin/kudu with pid 22617
I20260925 22:46:15.269265 22760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:46:15.268844 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.198.2:44559 (local address 127.0.0.1:35312)
0925 22:46:15.269072 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0925 22:46:15.269154 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.198.2:44559: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":143}
I20260925 22:46:15.275924 11032 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrWXm8X/build/debug/bin/kudu with pid 22230
I20260925 22:46:15.283718 11032 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260925 22:46:15.619974 11032 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260925 22:46:15.620110 11032 mini_ranger.cc:66] Stopping Ranger...
I20260925 22:46:15.966254 11032 mini_ranger.cc:68] Stopped Ranger
2026-09-25 22:46:15.966 UTC [21816] LOG: received smart shutdown request
2026-09-25 22:46:15.970 UTC [21816] LOG: background worker "logical replication launcher" (PID 21825) exited with exit code 1
2026-09-25 22:46:15.970 UTC [21820] LOG: shutting down
2026-09-25 22:46:16.135 UTC [21816] LOG: database system is shut down
2026-09-25T22:46:16Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (52224 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376259650642-11032-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 22:46:16 dist-test-slave-5vdr krb5kdc[22803](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 22:46:16 dist-test-slave-5vdr krb5kdc[22803](info): set up 2 sockets
Sep 25 22:46:16 dist-test-slave-5vdr krb5kdc[22803](info): commencing operation
krb5kdc: starting...
W20260925 22:46:17.312055 11032 mini_kdc.cc:121] Time spent starting KDC: real 1.033s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376259650642-11032-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376259650642-11032-0/krb5kdc/test-user.keytab.
Sep 25 22:46:17 dist-test-slave-5vdr krb5kdc[22803](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376377, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T22:46:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T22:46:17Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.10.198.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.10.198.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.10.198.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376259650642-11032-0/krb5kdc/rangeradmin_127.10.198.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.10.198.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376259650642-11032-0/krb5kdc/rangeradmin_127.10.198.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.10.198.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.10.198.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.10.198.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376259650642-11032-0/krb5kdc/rangerlookup_127.10.198.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.10.198.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376259650642-11032-0/krb5kdc/rangerlookup_127.10.198.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.10.198.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.10.198.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.10.198.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376259650642-11032-0/krb5kdc/HTTP_127.10.198.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.10.198.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376259650642-11032-0/krb5kdc/HTTP_127.10.198.20@KRBTEST.COM.keytab.
I20260925 22:46:17.499207 11032 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376259650642-11032-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskrWXm8X/build/debug/bin/postgres/pg_ctl -D /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376259650642-11032-0/minicluster-data/postgres -l logfile start
2026-09-25 22:46:20.161 UTC [22833] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-25 22:46:20.161 UTC [22833] LOG: listening on IPv4 address "127.10.198.20", port 57409
2026-09-25 22:46:20.165 UTC [22833] LOG: listening on Unix socket "/tmp/.s.PGSQL.57409"
2026-09-25 22:46:20.188 UTC [22836] LOG: database system was shut down at 2026-09-25 22:46:18 UTC
2026-09-25 22:46:20.192 UTC [22833] LOG: database system is ready to accept connections
I20260925 22:46:21.151899 11032 mini_postgres.cc:96] Postgres bound to 57409
2026-09-25 22:46:21.158 UTC [22844] FATAL: database "slave" does not exist
127.10.198.20:57409 - accepting connections
I20260925 22:46:21.159461 11032 mini_ranger.cc:161] Starting Ranger...
I20260925 22:46:21.176119 11032 mini_ranger.cc:84] Created miniranger Postgres user
I20260925 22:46:21.700376 11032 mini_ranger.cc:87] Created ranger Postgres database
I20260925 22:46:21.700601 11032 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376259650642-11032-0/minicluster-data/ranger-admin
2026-09-25 22:46:21,977 [I] DB FLAVOR :POSTGRES
2026-09-25 22:46:21,978 [I] --------- Verifying Ranger DB connection ---------
2026-09-25 22:46:21,978 [I] Checking connection..
2026-09-25 22:46:21,978 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskrWXm8X/build/debug/bin/postgresql.jar:/tmp/dist-test-taskrWXm8X/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.198.20:57409/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-25 22:46:22,244 [I] Checking connection passed.
2026-09-25 22:46:22,244 [I] --------- Verifying version history table ---------
2026-09-25 22:46:22,244 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskrWXm8X/build/debug/bin/postgresql.jar:/tmp/dist-test-taskrWXm8X/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.198.20:57409/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-25 22:46:22,500 [I] Table x_db_version_h does not exist in database ranger
2026-09-25 22:46:22,500 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskrWXm8X/build/debug/bin/postgresql.jar:/tmp/dist-test-taskrWXm8X/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.198.20:57409/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-25 22:46:22,776 [I] Table x_db_version_h does not exist in database ranger
2026-09-25 22:46:22,777 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-25 22:46:22,777 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskrWXm8X/build/debug/bin/postgresql.jar:/tmp/dist-test-taskrWXm8X/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.198.20:57409/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskrWXm8X/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-25 22:46:23.037 UTC [22908] WARNING: there is no transaction in progress
2026-09-25 22:46:23,044 [I] create_dbversion_catalog.sql file imported successfully
2026-09-25 22:46:23,044 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskrWXm8X/build/debug/bin/postgresql.jar:/tmp/dist-test-taskrWXm8X/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.198.20:57409/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-25 22:46:23,315 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-25 22:46:23,315 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-25 22:46:23,315 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskrWXm8X/build/debug/bin/postgresql.jar:/tmp/dist-test-taskrWXm8X/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.198.20:57409/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-25 22:46:23,570 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskrWXm8X/build/debug/bin/postgresql.jar:/tmp/dist-test-taskrWXm8X/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.198.20:57409/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-25 22:46:23,830 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskrWXm8X/build/debug/bin/postgresql.jar:/tmp/dist-test-taskrWXm8X/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.198.20:57409/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-5vdr.c.gcp-upstream.internal','N') ;"
2026-09-25 22:46:24,075 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskrWXm8X/build/debug/bin/postgresql.jar:/tmp/dist-test-taskrWXm8X/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.198.20:57409/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-25 22:46:24,359 [I] Table x_portal_user does not exist in database ranger
2026-09-25 22:46:24,359 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskrWXm8X/build/debug/bin/postgresql.jar:/tmp/dist-test-taskrWXm8X/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.198.20:57409/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-25 22:46:24,646 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-25 22:46:24,647 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskrWXm8X/build/debug/bin/postgresql.jar:/tmp/dist-test-taskrWXm8X/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.198.20:57409/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-25 22:46:24,901 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskrWXm8X/build/debug/bin/postgresql.jar:/tmp/dist-test-taskrWXm8X/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.198.20:57409/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-25 22:46:25,168 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-25 22:46:25,168 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskrWXm8X/build/debug/bin/postgresql.jar:/tmp/dist-test-taskrWXm8X/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.198.20:57409/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskrWXm8X/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-25 22:46:26.227 UTC [23016] WARNING: there is no transaction in progress
2026-09-25 22:46:26.245 UTC [23016] WARNING: there is no transaction in progress
2026-09-25 22:46:26.264 UTC [23016] WARNING: there is no transaction in progress
2026-09-25 22:46:26.280 UTC [23016] WARNING: there is no transaction in progress
2026-09-25 22:46:26.297 UTC [23016] WARNING: there is no transaction in progress
2026-09-25 22:46:26.316 UTC [23016] WARNING: there is no transaction in progress
2026-09-25 22:46:26.377 UTC [23016] WARNING: there is no transaction in progress
2026-09-25 22:46:26.385 UTC [23016] WARNING: there is no transaction in progress
2026-09-25 22:46:26.402 UTC [23016] WARNING: there is no transaction in progress
2026-09-25 22:46:26.414 UTC [23016] WARNING: there is no transaction in progress
2026-09-25 22:46:26.426 UTC [23016] WARNING: there is no transaction in progress
2026-09-25 22:46:26.443 UTC [23016] WARNING: there is no transaction in progress
2026-09-25 22:46:26.454 UTC [23016] WARNING: there is no transaction in progress
2026-09-25 22:46:26.462 UTC [23016] WARNING: there is no transaction in progress
2026-09-25 22:46:26.470 UTC [23016] WARNING: there is no transaction in progress
2026-09-25 22:46:26.478 UTC [23016] WARNING: there is no transaction in progress
2026-09-25 22:46:27,250 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-25 22:46:27,250 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskrWXm8X/build/debug/bin/postgresql.jar:/tmp/dist-test-taskrWXm8X/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.198.20:57409/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-25 22:46:27,520 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-25 22:46:27,521 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskrWXm8X/build/debug/bin/postgresql.jar:/tmp/dist-test-taskrWXm8X/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.198.20:57409/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-25 22:46:27,826 [I] Table x_portal_user already exists in database 'ranger'
2026-09-25 22:46:27,826 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskrWXm8X/build/debug/bin/postgresql.jar:/tmp/dist-test-taskrWXm8X/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.198.20:57409/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-25 22:46:28,078 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-25 22:46:28,078 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskrWXm8X/build/debug/bin/postgresql.jar:/tmp/dist-test-taskrWXm8X/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.198.20:57409/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-25 22:46:28,331 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskrWXm8X/build/debug/bin/postgresql.jar:/tmp/dist-test-taskrWXm8X/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.198.20:57409/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-25 22:46:28,605 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskrWXm8X/build/debug/bin/postgresql.jar:/tmp/dist-test-taskrWXm8X/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.198.20:57409/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-5vdr.c.gcp-upstream.internal';"
2026-09-25 22:46:28,866 [I] CORE_DB_SCHEMA import status has been updated
2026-09-25 22:46:28,867 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskrWXm8X/build/debug/bin/postgresql.jar:/tmp/dist-test-taskrWXm8X/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.198.20:57409/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-25 22:46:29,119 [I] DB_PATCHES have already been applied
I20260925 22:46:29.125747 11032 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376259650642-11032-0/minicluster-data/ranger-admin:/tmp/dist-test-taskrWXm8X/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskrWXm8X/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskrWXm8X/build/debug/bin/postgresql.jar:/tmp/dist-test-taskrWXm8X/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260925 22:46:29.125862 11032 mini_ranger.cc:193] Using host: 127.10.198.20
I20260925 22:46:29.129752 11032 mini_ranger.cc:227] Ranger admin URL: http://127.10.198.20:39709
Sep 25, 2026 10:46:29 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 25, 2026 10:46:29 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 25, 2026 10:46:29 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376259650642-11032-0/minicluster-data/ranger-admin/ews/webapp
Sep 25, 2026 10:46:29 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376259650642-11032-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 25, 2026 10:46:29 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376259650642-11032-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 25, 2026 10:46:30 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376259650642-11032-0/minicluster-data/ranger-admin/ews/webapp].
Sep 25, 2026 10:46:30 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260925 22:46:30.267120 11032 mini_ranger.cc:160] Time spent starting Ranger: real 9.108s user 0.001s sys 0.010s
Sep 25, 2026 10:46:53 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 25, 2026 10:46:53 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Sep 25, 2026 10:46:53 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 25, 2026 10:46:53 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 25, 2026 10:46:53 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 25, 2026 10:46:53 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 25, 2026 10:46:53 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 25, 2026 10:46:53 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 25, 2026 10:46:53 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 25, 2026 10:46:53 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 25, 2026 10:46:53 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 25, 2026 10:46:53 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 25, 2026 10:46:53 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 25, 2026 10:46:53 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 25, 2026 10:46:53 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 25, 2026 10:46:53 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 25, 2026 10:46:53 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 25, 2026 10:46:54 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260925 22:46:55.292945 11032 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.10.198.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.10.198.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.10.198.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376259650642-11032-0/krb5kdc/rangerkms_127.10.198.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.10.198.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376259650642-11032-0/krb5kdc/rangerkms_127.10.198.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.10.198.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.10.198.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.10.198.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376259650642-11032-0/krb5kdc/HTTP_127.10.198.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.10.198.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376259650642-11032-0/krb5kdc/HTTP_127.10.198.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 25 22:46:55 dist-test-slave-5vdr krb5kdc[22803](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376415, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260925 22:46:55.396078 11032 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260925 22:46:55.406869 11032 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260925 22:46:56.412703 11032 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260925 22:46:56.412827 11032 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376259650642-11032-0/minicluster-data/ranger-kms
I20260925 22:46:56.412851 11032 mini_ranger_kms.cc:227] Using postgres at 127.10.198.20:57409
2026-09-25 22:46:56,593 [I] DB FLAVOR :POSTGRES
2026-09-25 22:46:56,593 [I] --------- Verifying Ranger DB connection ---------
2026-09-25 22:46:56,593 [I] Checking connection
2026-09-25 22:46:56,593 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskrWXm8X/build/debug/bin/postgresql.jar:/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376259650642-11032-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.198.20:57409/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-25 22:46:56,844 [I] connection success
2026-09-25 22:46:56,844 [I] --------- Verifying Ranger DB tables ---------
2026-09-25 22:46:56,844 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-25 22:46:56,844 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskrWXm8X/build/debug/bin/postgresql.jar:/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376259650642-11032-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.198.20:57409/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-25 22:46:57,127 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-25 22:46:57,127 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-25 22:46:57,128 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-25 22:46:57,128 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskrWXm8X/build/debug/bin/postgresql.jar:/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376259650642-11032-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.198.20:57409/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376259650642-11032-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-25 22:46:57,418 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260925 22:46:58.030505 11032 mini_ranger_kms.cc:310] Created kms service
I20260925 22:46:58.257918 11032 mini_ranger_kms.cc:326] Created kudu user
I20260925 22:46:58.310055 11032 mini_ranger_kms.cc:343] Created rangerkms user
Sep 25 22:46:58 dist-test-slave-5vdr krb5kdc[22803](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376415, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.10.198.20@KRBTEST.COM
I20260925 22:46:58.738258 11032 mini_ranger_kms.cc:384] Added ranger policy
I20260925 22:46:58.738440 11032 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376259650642-11032-0/minicluster-data/ranger-kms:/tmp/dist-test-taskrWXm8X/build/debug/bin/postgresql.jar:/tmp/dist-test-taskrWXm8X/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskrWXm8X/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260925 22:46:58.738469 11032 mini_ranger_kms.cc:240] Using host: 127.10.198.20
I20260925 22:46:58.743503 11032 mini_ranger_kms.cc:276] Ranger KMS PID: 23197
I20260925 22:46:58.743623 11032 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.10.198.20:47825
Sep 25, 2026 10:47:00 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 25, 2026 10:47:00 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 25, 2026 10:47:00 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376259650642-11032-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 25, 2026 10:47:00 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376259650642-11032-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 25, 2026 10:47:00 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376259650642-11032-0/minicluster-data/ranger-kms/ews/webapp].
Sep 25, 2026 10:47:00 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260925 22:47:01.071762 11032 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.676s user 0.010s sys 0.154s
I20260925 22:47:01.071928 11032 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260925 22:47:01.072016 11032 mini_ranger_kms.cc:401] 127.10.198.20:47825/kms/v1/keys
Sep 25 22:47:01 dist-test-slave-5vdr krb5kdc[22803](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790376421, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.10.198.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 25 22:47:01 dist-test-slave-5vdr krb5kdc[22803](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376421, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.10.198.20@KRBTEST.COM for HTTP/127.10.198.20@KRBTEST.COM
Sep 25 22:47:02 dist-test-slave-5vdr krb5kdc[22803](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376421, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.10.198.20@KRBTEST.COM for HTTP/127.10.198.20@KRBTEST.COM
Sep 25 22:47:04 dist-test-slave-5vdr krb5kdc[22803](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376424, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.10.198.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.198.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.198.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376259650642-11032-0/krb5kdc/kudu_127.10.198.62.keytab.
Entry for principal kudu/127.10.198.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376259650642-11032-0/krb5kdc/kudu_127.10.198.62.keytab.
I20260925 22:47:04.635525 11032 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrWXm8X/build/debug/bin/kudu
/tmp/dist-test-taskrWXm8X/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376259650642-11032-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376259650642-11032-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376259650642-11032-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376259650642-11032-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.198.62:38359
--webserver_interface=127.10.198.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376259650642-11032-0/krb5kdc/kudu_127.10.198.62.keytab
--principal=kudu/127.10.198.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.198.20:34721
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.198.62:38359
--ranger_config_path=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376259650642-11032-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.10.198.20:47825/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376259650642-11032-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376259650642-11032-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376259650642-11032-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:47:04.759661 23247 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:47:04.760123 23247 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:47:04.760217 23247 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:47:04.764614 23247 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 22:47:04.764743 23247 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:47:04.764780 23247 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:47:04.764822 23247 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 22:47:04.764858 23247 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 22:47:04.771272 23247 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.198.20:34721
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376259650642-11032-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376259650642-11032-0/minicluster-data/master-0/wal
--ranger_kms_url=127.10.198.20:47825/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.10.198.62:38359
--ranger_config_path=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376259650642-11032-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376259650642-11032-0/krb5kdc/kudu_127.10.198.62.keytab
--principal=kudu/127.10.198.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.198.62:38359
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376259650642-11032-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.198.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23247
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376259650642-11032-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:09:16 UTC on ccycloud.quasar-afmhyw.root.comops.site
build id 4673379
I20260925 22:47:04.772627 23247 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:47:04.773802 23247 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:47:04.783414 23252 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:47:04.783416 23255 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:47:04.783608 23247 server_base.cc:1034] running on GCE node
W20260925 22:47:04.783419 23253 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:47:04.784425 23247 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:47:04.785825 23247 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:47:04.787050 23247 hybrid_clock.cc:648] HybridClock initialized: now 1790376424787017 us; error 61 us; skew 500 ppm
Sep 25 22:47:04 dist-test-slave-5vdr krb5kdc[22803](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376424, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.198.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:47:04.790338 23247 init.cc:377] Logged in from keytab as kudu/127.10.198.62@KRBTEST.COM (short username kudu)
I20260925 22:47:04.791486 23247 webserver.cc:460] Webserver started at http://127.10.198.62:40291/ using document root <none> and password file <none>
I20260925 22:47:04.792204 23247 fs_manager.cc:359] Metadata directory not provided
I20260925 22:47:04.792299 23247 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:47:04.792551 23247 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 25 22:47:04 dist-test-slave-5vdr krb5kdc[22803](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376424, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.198.62@KRBTEST.COM for HTTP/127.10.198.20@KRBTEST.COM
I20260925 22:47:04.844686 23247 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376259650642-11032-0/minicluster-data/master-0/data/instance:
uuid: "ec9d2e7ae7f1418caff74bbf04252560"
format_stamp: "Formatted at 2026-09-25 22:47:04 on dist-test-slave-5vdr"
server_key: "a1975049dcac9369963f28f867b5fcba"
server_key_iv: "4f9767831b7e71bde5df4b5a3c46e8db"
server_key_version: "kuduclusterkey@0"
I20260925 22:47:04.845391 23247 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376259650642-11032-0/minicluster-data/master-0/wal/instance:
uuid: "ec9d2e7ae7f1418caff74bbf04252560"
format_stamp: "Formatted at 2026-09-25 22:47:04 on dist-test-slave-5vdr"
server_key: "a1975049dcac9369963f28f867b5fcba"
server_key_iv: "4f9767831b7e71bde5df4b5a3c46e8db"
server_key_version: "kuduclusterkey@0"
I20260925 22:47:04.849866 23247 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260925 22:47:04.865867 23262 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:47:04.867411 23247 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260925 22:47:04.867594 23247 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376259650642-11032-0/minicluster-data/master-0/data,/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376259650642-11032-0/minicluster-data/master-0/wal
uuid: "ec9d2e7ae7f1418caff74bbf04252560"
format_stamp: "Formatted at 2026-09-25 22:47:04 on dist-test-slave-5vdr"
server_key: "a1975049dcac9369963f28f867b5fcba"
server_key_iv: "4f9767831b7e71bde5df4b5a3c46e8db"
server_key_version: "kuduclusterkey@0"
I20260925 22:47:04.867748 23247 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376259650642-11032-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376259650642-11032-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376259650642-11032-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:47:04.894634 23247 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:47:04.898108 23247 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:47:04.898370 23247 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:47:04.907018 23247 rpc_server.cc:307] RPC server started. Bound to: 127.10.198.62:38359
I20260925 22:47:04.907020 23314 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.198.62:38359 every 8 connection(s)
I20260925 22:47:04.908303 23247 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376259650642-11032-0/minicluster-data/master-0/data/info.pb
I20260925 22:47:04.911417 23315 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:47:04.914708 11032 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrWXm8X/build/debug/bin/kudu as pid 23247
I20260925 22:47:04.914868 11032 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376259650642-11032-0/minicluster-data/master-0/wal/instance
I20260925 22:47:04.915202 11032 external_mini_cluster.cc:1428] Setting key 8bbd7a63f686b943bc1502d24d9fd690
I20260925 22:47:04.919248 23315 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ec9d2e7ae7f1418caff74bbf04252560: Bootstrap starting.
I20260925 22:47:04.922173 23315 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ec9d2e7ae7f1418caff74bbf04252560: Neither blocks nor log segments found. Creating new log.
I20260925 22:47:04.923285 23315 log.cc:824] T 00000000000000000000000000000000 P ec9d2e7ae7f1418caff74bbf04252560: Log is configured to *not* fsync() on all Append() calls
I20260925 22:47:04.926192 23315 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ec9d2e7ae7f1418caff74bbf04252560: No bootstrap required, opened a new log
Sep 25 22:47:04 dist-test-slave-5vdr krb5kdc[22803](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376424, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.198.62@KRBTEST.COM
I20260925 22:47:04.931030 23315 raft_consensus.cc:348] T 00000000000000000000000000000000 P ec9d2e7ae7f1418caff74bbf04252560 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec9d2e7ae7f1418caff74bbf04252560" member_type: VOTER last_known_addr { host: "127.10.198.62" port: 38359 } }
I20260925 22:47:04.931409 23315 raft_consensus.cc:374] T 00000000000000000000000000000000 P ec9d2e7ae7f1418caff74bbf04252560 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:47:04.931519 23315 raft_consensus.cc:729] T 00000000000000000000000000000000 P ec9d2e7ae7f1418caff74bbf04252560 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ec9d2e7ae7f1418caff74bbf04252560, State: Initialized, Role: FOLLOWER
I20260925 22:47:04.932178 23315 consensus_queue.cc:260] T 00000000000000000000000000000000 P ec9d2e7ae7f1418caff74bbf04252560 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec9d2e7ae7f1418caff74bbf04252560" member_type: VOTER last_known_addr { host: "127.10.198.62" port: 38359 } }
I20260925 22:47:04.932364 23315 raft_consensus.cc:388] T 00000000000000000000000000000000 P ec9d2e7ae7f1418caff74bbf04252560 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 22:47:04.932474 23315 raft_consensus.cc:482] T 00000000000000000000000000000000 P ec9d2e7ae7f1418caff74bbf04252560 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 22:47:04.932582 23315 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ec9d2e7ae7f1418caff74bbf04252560 [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:47:04.933979 23315 raft_consensus.cc:504] T 00000000000000000000000000000000 P ec9d2e7ae7f1418caff74bbf04252560 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec9d2e7ae7f1418caff74bbf04252560" member_type: VOTER last_known_addr { host: "127.10.198.62" port: 38359 } }
I20260925 22:47:04.934404 23315 leader_election.cc:302] T 00000000000000000000000000000000 P ec9d2e7ae7f1418caff74bbf04252560 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ec9d2e7ae7f1418caff74bbf04252560; no voters:
I20260925 22:47:04.934794 23315 leader_election.cc:288] T 00000000000000000000000000000000 P ec9d2e7ae7f1418caff74bbf04252560 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 22:47:04.935031 23320 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ec9d2e7ae7f1418caff74bbf04252560 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 22:47:04.935317 23320 raft_consensus.cc:686] T 00000000000000000000000000000000 P ec9d2e7ae7f1418caff74bbf04252560 [term 1 LEADER]: Becoming Leader. State: Replica: ec9d2e7ae7f1418caff74bbf04252560, State: Running, Role: LEADER
I20260925 22:47:04.935417 23318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:47:04.916899 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60722 (local address 127.10.198.62:38359)
0925 22:47:04.917436 (+ 537us) server_negotiation.cc:206] Beginning negotiation
0925 22:47:04.917456 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0925 22:47:04.917507 (+ 51us) server_negotiation.cc:407] Connection header received
0925 22:47:04.918419 (+ 912us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:47:04.918447 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:47:04.918931 (+ 484us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:47:04.920061 (+ 1130us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:47:04.920606 (+ 545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:47:04.922234 (+ 1628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:47:04.923640 (+ 1406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:47:04.924509 (+ 869us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:47:04.924775 (+ 266us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:47:04.928385 (+ 3610us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:47:04.928395 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:47:04.928419 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:47:04.928465 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:47:04.930757 (+ 2292us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:47:04.931515 (+ 758us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:47:04.931522 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:47:04.931531 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:47:04.931709 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:47:04.932106 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:47:04.932109 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:47:04.932111 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:47:04.932744 (+ 633us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:47:04.933729 (+ 985us) server_negotiation.cc:1035] Waiting for connection context
0925 22:47:04.933915 (+ 186us) server_negotiation.cc:299] Negotiation successful
0925 22:47:04.934264 (+ 349us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":367,"thread_start_us":171,"threads_started":1}
I20260925 22:47:04.935722 23320 consensus_queue.cc:237] T 00000000000000000000000000000000 P ec9d2e7ae7f1418caff74bbf04252560 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec9d2e7ae7f1418caff74bbf04252560" member_type: VOTER last_known_addr { host: "127.10.198.62" port: 38359 } }
I20260925 22:47:04.936280 23315 sys_catalog.cc:564] T 00000000000000000000000000000000 P ec9d2e7ae7f1418caff74bbf04252560 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 22:47:04.937709 23320 sys_catalog.cc:455] T 00000000000000000000000000000000 P ec9d2e7ae7f1418caff74bbf04252560 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ec9d2e7ae7f1418caff74bbf04252560. Latest consensus state: current_term: 1 leader_uuid: "ec9d2e7ae7f1418caff74bbf04252560" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec9d2e7ae7f1418caff74bbf04252560" member_type: VOTER last_known_addr { host: "127.10.198.62" port: 38359 } } }
I20260925 22:47:04.937839 23320 sys_catalog.cc:458] T 00000000000000000000000000000000 P ec9d2e7ae7f1418caff74bbf04252560 [sys.catalog]: This master's current role is: LEADER
I20260925 22:47:04.938542 23322 sys_catalog.cc:455] T 00000000000000000000000000000000 P ec9d2e7ae7f1418caff74bbf04252560 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ec9d2e7ae7f1418caff74bbf04252560" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec9d2e7ae7f1418caff74bbf04252560" member_type: VOTER last_known_addr { host: "127.10.198.62" port: 38359 } } }
I20260925 22:47:04.938781 23322 sys_catalog.cc:458] T 00000000000000000000000000000000 P ec9d2e7ae7f1418caff74bbf04252560 [sys.catalog]: This master's current role is: LEADER
I20260925 22:47:04.939055 23315 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376259650642-11032-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260925 22:47:04.940562 23329 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 22:47:04.947921 23329 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 22:47:04.956265 23329 catalog_manager.cc:1347] Generated new cluster ID: 21f17c2efe554409a68eae6c72717ccc
I20260925 22:47:04.957633 23329 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 22:47:04.986991 23329 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 22:47:04.988175 23329 catalog_manager.cc:1504] Loading token signing keys...
I20260925 22:47:04.998948 23329 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ec9d2e7ae7f1418caff74bbf04252560: Generated new TSK 0
I20260925 22:47:05.000072 23329 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.10.198.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.198.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.198.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376259650642-11032-0/krb5kdc/kudu_127.10.198.1.keytab.
Entry for principal kudu/127.10.198.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376259650642-11032-0/krb5kdc/kudu_127.10.198.1.keytab.
I20260925 22:47:05.804488 11032 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrWXm8X/build/debug/bin/kudu
/tmp/dist-test-taskrWXm8X/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376259650642-11032-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376259650642-11032-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376259650642-11032-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376259650642-11032-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.10.198.20:47825/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.198.1:0
--local_ip_for_outbound_sockets=127.10.198.1
--webserver_interface=127.10.198.1
--webserver_port=0
--tserver_master_addrs=127.10.198.62:38359
--keytab_file=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376259650642-11032-0/krb5kdc/kudu_127.10.198.1.keytab
--principal=kudu/127.10.198.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.198.20:34721
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376259650642-11032-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376259650642-11032-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376259650642-11032-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:47:05.993198 23360 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:47:05.994091 23360 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:47:05.994232 23360 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:47:06.002290 23360 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:47:06.002494 23360 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:47:06.002664 23360 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.198.1
I20260925 22:47:06.012393 23360 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.198.20:34721
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376259650642-11032-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376259650642-11032-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.10.198.20:47825/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376259650642-11032-0/krb5kdc/kudu_127.10.198.1.keytab
--principal=kudu/127.10.198.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.198.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376259650642-11032-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.198.1
--webserver_port=0
--tserver_master_addrs=127.10.198.62:38359
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23360
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.198.1
--log_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376259650642-11032-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:09:16 UTC on ccycloud.quasar-afmhyw.root.comops.site
build id 4673379
I20260925 22:47:06.014111 23360 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:47:06.015785 23360 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:47:06.031876 23368 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:47:06.032470 23370 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:47:06.031993 23367 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:47:06.033332 23360 server_base.cc:1034] running on GCE node
I20260925 22:47:06.034160 23360 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:47:06.035122 23360 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:47:06.038442 23360 hybrid_clock.cc:648] HybridClock initialized: now 1790376426038388 us; error 54 us; skew 500 ppm
Sep 25 22:47:06 dist-test-slave-5vdr krb5kdc[22803](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376426, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.198.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:47:06.042634 23360 init.cc:377] Logged in from keytab as kudu/127.10.198.1@KRBTEST.COM (short username kudu)
I20260925 22:47:06.043989 23360 webserver.cc:460] Webserver started at http://127.10.198.1:39569/ using document root <none> and password file <none>
I20260925 22:47:06.044756 23360 fs_manager.cc:359] Metadata directory not provided
I20260925 22:47:06.044857 23360 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:47:06.045095 23360 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 25 22:47:06 dist-test-slave-5vdr krb5kdc[22803](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376426, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.198.1@KRBTEST.COM for HTTP/127.10.198.20@KRBTEST.COM
I20260925 22:47:06.067310 23360 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376259650642-11032-0/minicluster-data/ts-0/data/instance:
uuid: "fea07c2152bc490580f2782c1319a37d"
format_stamp: "Formatted at 2026-09-25 22:47:06 on dist-test-slave-5vdr"
server_key: "69fda9a198d8db7f4346722c1ab2b01e"
server_key_iv: "bb662cddd2d49de55f0e9395f73a1741"
server_key_version: "kuduclusterkey@0"
I20260925 22:47:06.068406 23360 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376259650642-11032-0/minicluster-data/ts-0/wal/instance:
uuid: "fea07c2152bc490580f2782c1319a37d"
format_stamp: "Formatted at 2026-09-25 22:47:06 on dist-test-slave-5vdr"
server_key: "69fda9a198d8db7f4346722c1ab2b01e"
server_key_iv: "bb662cddd2d49de55f0e9395f73a1741"
server_key_version: "kuduclusterkey@0"
I20260925 22:47:06.075067 23360 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260925 22:47:06.102383 23377 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:47:06.104678 23360 fs_manager.cc:725] Time spent opening block manager: real 0.008s user 0.002s sys 0.000s
I20260925 22:47:06.104868 23360 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376259650642-11032-0/minicluster-data/ts-0/data,/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376259650642-11032-0/minicluster-data/ts-0/wal
uuid: "fea07c2152bc490580f2782c1319a37d"
format_stamp: "Formatted at 2026-09-25 22:47:06 on dist-test-slave-5vdr"
server_key: "69fda9a198d8db7f4346722c1ab2b01e"
server_key_iv: "bb662cddd2d49de55f0e9395f73a1741"
server_key_version: "kuduclusterkey@0"
I20260925 22:47:06.105019 23360 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376259650642-11032-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376259650642-11032-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376259650642-11032-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:47:06.127197 23360 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:47:06.131919 23360 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:47:06.132236 23360 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:47:06.133105 23360 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 22:47:06.134425 23360 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:47:06.134557 23360 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:47:06.134702 23360 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:47:06.134785 23360 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 22:47:06.149618 23360 rpc_server.cc:307] RPC server started. Bound to: 127.10.198.1:36413
I20260925 22:47:06.151078 23360 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376259650642-11032-0/minicluster-data/ts-0/data/info.pb
I20260925 22:47:06.153473 23492 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.198.1:36413 every 8 connection(s)
I20260925 22:47:06.154384 11032 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrWXm8X/build/debug/bin/kudu as pid 23360
I20260925 22:47:06.154507 11032 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376259650642-11032-0/minicluster-data/ts-0/wal/instance
I20260925 22:47:06.154832 11032 external_mini_cluster.cc:1428] Setting key 43d7838bb2f2f155696c580630989a34
Sep 25 22:47:06 dist-test-slave-5vdr krb5kdc[22803](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376426, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.198.1@KRBTEST.COM for kudu/127.10.198.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.10.198.2@KRBTEST.COM; defaulting to no policy
I20260925 22:47:06.178607 23496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:47:06.157883 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.198.1:43493 (local address 127.10.198.62:38359)
0925 22:47:06.160214 (+ 2331us) server_negotiation.cc:206] Beginning negotiation
0925 22:47:06.160219 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:47:06.160247 (+ 28us) server_negotiation.cc:407] Connection header received
0925 22:47:06.160294 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:47:06.160297 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:47:06.160348 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:47:06.160441 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:47:06.162205 (+ 1764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:47:06.163087 (+ 882us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:47:06.165575 (+ 2488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:47:06.166039 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:47:06.166184 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:47:06.173036 (+ 6852us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:47:06.173045 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:47:06.173049 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:47:06.173096 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:47:06.175542 (+ 2446us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:47:06.176456 (+ 914us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:47:06.176460 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:47:06.176462 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:47:06.176579 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:47:06.177073 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:47:06.177078 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:47:06.177080 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:47:06.177360 (+ 280us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:47:06.177474 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0925 22:47:06.178187 (+ 713us) server_negotiation.cc:299] Negotiation successful
0925 22:47:06.178395 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2234,"thread_start_us":110,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.198.2@KRBTEST.COM" created.
I20260925 22:47:06.179673 23495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:47:06.156633 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.198.62:38359 (local address 127.10.198.1:43493)
0925 22:47:06.158498 (+ 1865us) negotiation.cc:107] Waiting for socket to connect
0925 22:47:06.158570 (+ 72us) client_negotiation.cc:174] Beginning negotiation
0925 22:47:06.159667 (+ 1097us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:47:06.160796 (+ 1129us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:47:06.160808 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:47:06.161251 (+ 443us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:47:06.161983 (+ 732us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:47:06.162006 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:47:06.163301 (+ 1295us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:47:06.163305 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:47:06.165302 (+ 1997us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:47:06.165326 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:47:06.166446 (+ 1120us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:47:06.166451 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:47:06.166742 (+ 291us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:47:06.167650 (+ 908us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:47:06.167679 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:47:06.172673 (+ 4994us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:47:06.175828 (+ 3155us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:47:06.175878 (+ 50us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:47:06.175905 (+ 27us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:47:06.176286 (+ 381us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:47:06.176688 (+ 402us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:47:06.176693 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:47:06.176695 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:47:06.176935 (+ 240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:47:06.177488 (+ 553us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:47:06.177498 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:47:06.177886 (+ 388us) client_negotiation.cc:769] Sending connection context
0925 22:47:06.178108 (+ 222us) client_negotiation.cc:240] Negotiation successful
0925 22:47:06.178488 (+ 380us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1602,"thread_start_us":179,"threads_started":1}
I20260925 22:47:06.183399 23493 heartbeater.cc:347] Connected to a master server at 127.10.198.62:38359
I20260925 22:47:06.183718 23493 heartbeater.cc:464] Registering TS with master...
I20260925 22:47:06.184497 23493 heartbeater.cc:511] Master 127.10.198.62:38359 requested a full tablet report, sending...
I20260925 22:47:06.186523 23279 ts_manager.cc:194] Registered new tserver with Master: fea07c2152bc490580f2782c1319a37d (127.10.198.1:36413)
I20260925 22:47:06.188215 23279 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.198.1@KRBTEST.COM'} at 127.10.198.1:43493
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.198.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376259650642-11032-0/krb5kdc/kudu_127.10.198.2.keytab.
Entry for principal kudu/127.10.198.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376259650642-11032-0/krb5kdc/kudu_127.10.198.2.keytab.
I20260925 22:47:06.208391 11032 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrWXm8X/build/debug/bin/kudu
/tmp/dist-test-taskrWXm8X/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376259650642-11032-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376259650642-11032-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376259650642-11032-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376259650642-11032-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.10.198.20:47825/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.198.2:0
--local_ip_for_outbound_sockets=127.10.198.2
--webserver_interface=127.10.198.2
--webserver_port=0
--tserver_master_addrs=127.10.198.62:38359
--keytab_file=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376259650642-11032-0/krb5kdc/kudu_127.10.198.2.keytab
--principal=kudu/127.10.198.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.198.20:34721
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376259650642-11032-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376259650642-11032-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376259650642-11032-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 25 22:47:06 dist-test-slave-5vdr krb5kdc[22803](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790376426, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.198.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260925 22:47:06.369151 23499 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:47:06.369498 23499 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:47:06.369611 23499 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:47:06.374748 23499 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:47:06.374850 23499 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:47:06.374953 23499 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.198.2
I20260925 22:47:06.381299 23499 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.198.20:34721
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376259650642-11032-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376259650642-11032-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.10.198.20:47825/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376259650642-11032-0/krb5kdc/kudu_127.10.198.2.keytab
--principal=kudu/127.10.198.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.198.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376259650642-11032-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.198.2
--webserver_port=0
--tserver_master_addrs=127.10.198.62:38359
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23499
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.198.2
--log_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376259650642-11032-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:09:16 UTC on ccycloud.quasar-afmhyw.root.comops.site
build id 4673379
I20260925 22:47:06.382596 23499 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:47:06.383739 23499 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:47:06.394556 23508 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:47:06.394556 23505 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:47:06.394995 23506 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:47:06.396096 23499 server_base.cc:1034] running on GCE node
I20260925 22:47:06.396643 23499 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:47:06.397454 23499 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:47:06.398682 23499 hybrid_clock.cc:648] HybridClock initialized: now 1790376426398644 us; error 59 us; skew 500 ppm
Sep 25 22:47:06 dist-test-slave-5vdr krb5kdc[22803](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376426, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.198.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:47:06.403158 23499 init.cc:377] Logged in from keytab as kudu/127.10.198.2@KRBTEST.COM (short username kudu)
I20260925 22:47:06.404824 23499 webserver.cc:460] Webserver started at http://127.10.198.2:35411/ using document root <none> and password file <none>
I20260925 22:47:06.405640 23499 fs_manager.cc:359] Metadata directory not provided
I20260925 22:47:06.405756 23499 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:47:06.406015 23499 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 25 22:47:06 dist-test-slave-5vdr krb5kdc[22803](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376426, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.198.2@KRBTEST.COM for HTTP/127.10.198.20@KRBTEST.COM
I20260925 22:47:06.426105 23499 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376259650642-11032-0/minicluster-data/ts-1/data/instance:
uuid: "01b6959223424811be44029b72bdac12"
format_stamp: "Formatted at 2026-09-25 22:47:06 on dist-test-slave-5vdr"
server_key: "647135544f5962dd2b6a6aa272c19ecb"
server_key_iv: "a36a0d6841e64ab5b2426fcfa2030bbc"
server_key_version: "kuduclusterkey@0"
I20260925 22:47:06.426946 23499 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376259650642-11032-0/minicluster-data/ts-1/wal/instance:
uuid: "01b6959223424811be44029b72bdac12"
format_stamp: "Formatted at 2026-09-25 22:47:06 on dist-test-slave-5vdr"
server_key: "647135544f5962dd2b6a6aa272c19ecb"
server_key_iv: "a36a0d6841e64ab5b2426fcfa2030bbc"
server_key_version: "kuduclusterkey@0"
I20260925 22:47:06.432825 23499 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260925 22:47:06.458797 23515 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:47:06.461016 23499 fs_manager.cc:725] Time spent opening block manager: real 0.007s user 0.004s sys 0.000s
I20260925 22:47:06.461220 23499 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376259650642-11032-0/minicluster-data/ts-1/data,/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376259650642-11032-0/minicluster-data/ts-1/wal
uuid: "01b6959223424811be44029b72bdac12"
format_stamp: "Formatted at 2026-09-25 22:47:06 on dist-test-slave-5vdr"
server_key: "647135544f5962dd2b6a6aa272c19ecb"
server_key_iv: "a36a0d6841e64ab5b2426fcfa2030bbc"
server_key_version: "kuduclusterkey@0"
I20260925 22:47:06.461436 23499 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376259650642-11032-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376259650642-11032-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376259650642-11032-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:47:06.490022 23499 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:47:06.494351 23499 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:47:06.494570 23499 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:47:06.495337 23499 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 22:47:06.496523 23499 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:47:06.496595 23499 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:47:06.496663 23499 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:47:06.496694 23499 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 22:47:06.509630 23499 rpc_server.cc:307] RPC server started. Bound to: 127.10.198.2:34763
I20260925 22:47:06.509809 23628 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.198.2:34763 every 8 connection(s)
I20260925 22:47:06.512557 23499 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376259650642-11032-0/minicluster-data/ts-1/data/info.pb
I20260925 22:47:06.522488 11032 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrWXm8X/build/debug/bin/kudu as pid 23499
I20260925 22:47:06.522639 11032 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376259650642-11032-0/minicluster-data/ts-1/wal/instance
I20260925 22:47:06.522984 11032 external_mini_cluster.cc:1428] Setting key 4e5b1f7e657348f70140408858ebb4e1
Sep 25 22:47:06 dist-test-slave-5vdr krb5kdc[22803](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376426, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.198.2@KRBTEST.COM for kudu/127.10.198.62@KRBTEST.COM
I20260925 22:47:06.538360 23496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:47:06.518197 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.198.2:43651 (local address 127.10.198.62:38359)
0925 22:47:06.518352 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0925 22:47:06.518358 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 22:47:06.519353 (+ 995us) server_negotiation.cc:407] Connection header received
0925 22:47:06.520781 (+ 1428us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:47:06.520786 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:47:06.520883 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:47:06.520953 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:47:06.522945 (+ 1992us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:47:06.523796 (+ 851us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:47:06.529126 (+ 5330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:47:06.529540 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:47:06.529641 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:47:06.533207 (+ 3566us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:47:06.533225 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:47:06.533228 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:47:06.533264 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:47:06.535144 (+ 1880us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:47:06.536014 (+ 870us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:47:06.536018 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:47:06.536020 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:47:06.536157 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:47:06.536648 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:47:06.536652 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:47:06.536654 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:47:06.536954 (+ 300us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:47:06.537046 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0925 22:47:06.537971 (+ 925us) server_negotiation.cc:299] Negotiation successful
0925 22:47:06.538160 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260925 22:47:06.540273 23631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:47:06.516906 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.198.62:38359 (local address 127.10.198.2:43651)
0925 22:47:06.519129 (+ 2223us) negotiation.cc:107] Waiting for socket to connect
0925 22:47:06.519202 (+ 73us) client_negotiation.cc:174] Beginning negotiation
0925 22:47:06.520460 (+ 1258us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:47:06.521486 (+ 1026us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:47:06.521497 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:47:06.521919 (+ 422us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:47:06.522718 (+ 799us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:47:06.522743 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:47:06.527153 (+ 4410us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:47:06.527157 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:47:06.528885 (+ 1728us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:47:06.528910 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:47:06.529915 (+ 1005us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:47:06.529919 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:47:06.530164 (+ 245us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:47:06.531004 (+ 840us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:47:06.531029 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:47:06.532918 (+ 1889us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:47:06.535381 (+ 2463us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:47:06.535389 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:47:06.535411 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:47:06.535822 (+ 411us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:47:06.536313 (+ 491us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:47:06.536316 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:47:06.536318 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:47:06.536532 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:47:06.537192 (+ 660us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:47:06.537200 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:47:06.537623 (+ 423us) client_negotiation.cc:769] Sending connection context
0925 22:47:06.538650 (+ 1027us) client_negotiation.cc:240] Negotiation successful
0925 22:47:06.539080 (+ 430us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1973,"thread_start_us":215,"threads_started":1}
I20260925 22:47:06.541803 23629 heartbeater.cc:347] Connected to a master server at 127.10.198.62:38359
I20260925 22:47:06.542090 23629 heartbeater.cc:464] Registering TS with master...
I20260925 22:47:06.542725 23629 heartbeater.cc:511] Master 127.10.198.62:38359 requested a full tablet report, sending...
I20260925 22:47:06.544006 23279 ts_manager.cc:194] Registered new tserver with Master: 01b6959223424811be44029b72bdac12 (127.10.198.2:34763)
I20260925 22:47:06.544771 23279 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.198.2@KRBTEST.COM'} at 127.10.198.2:43651
WARNING: no policy specified for kudu/127.10.198.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.198.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.198.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376259650642-11032-0/krb5kdc/kudu_127.10.198.3.keytab.
Entry for principal kudu/127.10.198.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376259650642-11032-0/krb5kdc/kudu_127.10.198.3.keytab.
I20260925 22:47:06.571153 11032 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrWXm8X/build/debug/bin/kudu
/tmp/dist-test-taskrWXm8X/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376259650642-11032-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376259650642-11032-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376259650642-11032-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376259650642-11032-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.10.198.20:47825/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.198.3:0
--local_ip_for_outbound_sockets=127.10.198.3
--webserver_interface=127.10.198.3
--webserver_port=0
--tserver_master_addrs=127.10.198.62:38359
--keytab_file=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376259650642-11032-0/krb5kdc/kudu_127.10.198.3.keytab
--principal=kudu/127.10.198.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.198.20:34721
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376259650642-11032-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376259650642-11032-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376259650642-11032-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:47:06.745505 23634 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:47:06.745939 23634 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:47:06.746004 23634 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:47:06.753682 23634 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:47:06.753804 23634 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:47:06.753899 23634 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.198.3
I20260925 22:47:06.762419 23634 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.198.20:34721
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376259650642-11032-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376259650642-11032-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.10.198.20:47825/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376259650642-11032-0/krb5kdc/kudu_127.10.198.3.keytab
--principal=kudu/127.10.198.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.198.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376259650642-11032-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.198.3
--webserver_port=0
--tserver_master_addrs=127.10.198.62:38359
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23634
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.198.3
--log_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376259650642-11032-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:09:16 UTC on ccycloud.quasar-afmhyw.root.comops.site
build id 4673379
I20260925 22:47:06.763957 23634 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:47:06.765322 23634 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 22:47:06.779922 23634 server_base.cc:1034] running on GCE node
W20260925 22:47:06.779556 23640 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:47:06.780488 23639 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:47:06.781034 23642 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:47:06.781620 23634 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:47:06.782528 23634 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:47:06.783704 23634 hybrid_clock.cc:648] HybridClock initialized: now 1790376426783636 us; error 90 us; skew 500 ppm
Sep 25 22:47:06 dist-test-slave-5vdr krb5kdc[22803](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376426, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.198.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:47:06.787482 23634 init.cc:377] Logged in from keytab as kudu/127.10.198.3@KRBTEST.COM (short username kudu)
I20260925 22:47:06.788954 23634 webserver.cc:460] Webserver started at http://127.10.198.3:39873/ using document root <none> and password file <none>
I20260925 22:47:06.789767 23634 fs_manager.cc:359] Metadata directory not provided
I20260925 22:47:06.790012 23634 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:47:06.790316 23634 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 25 22:47:06 dist-test-slave-5vdr krb5kdc[22803](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376426, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.198.3@KRBTEST.COM for HTTP/127.10.198.20@KRBTEST.COM
I20260925 22:47:06.812076 23634 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376259650642-11032-0/minicluster-data/ts-2/data/instance:
uuid: "5196eab08c014dd48322cc60aa5b54bf"
format_stamp: "Formatted at 2026-09-25 22:47:06 on dist-test-slave-5vdr"
server_key: "5e61f155fb57bd9de22fd7a5e2808f5c"
server_key_iv: "94a4bcdac39dc112f9180932adf16430"
server_key_version: "kuduclusterkey@0"
I20260925 22:47:06.814198 23634 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376259650642-11032-0/minicluster-data/ts-2/wal/instance:
uuid: "5196eab08c014dd48322cc60aa5b54bf"
format_stamp: "Formatted at 2026-09-25 22:47:06 on dist-test-slave-5vdr"
server_key: "5e61f155fb57bd9de22fd7a5e2808f5c"
server_key_iv: "94a4bcdac39dc112f9180932adf16430"
server_key_version: "kuduclusterkey@0"
I20260925 22:47:06.820168 23634 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
Sep 25 22:47:06 dist-test-slave-5vdr krb5kdc[22803](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376426, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.198.62@KRBTEST.COM for HTTP/127.10.198.20@KRBTEST.COM
I20260925 22:47:06.852581 23649 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:47:06.854377 23634 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260925 22:47:06.854532 23634 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376259650642-11032-0/minicluster-data/ts-2/data,/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376259650642-11032-0/minicluster-data/ts-2/wal
uuid: "5196eab08c014dd48322cc60aa5b54bf"
format_stamp: "Formatted at 2026-09-25 22:47:06 on dist-test-slave-5vdr"
server_key: "5e61f155fb57bd9de22fd7a5e2808f5c"
server_key_iv: "94a4bcdac39dc112f9180932adf16430"
server_key_version: "kuduclusterkey@0"
I20260925 22:47:06.854673 23634 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376259650642-11032-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376259650642-11032-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376259650642-11032-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:47:06.896266 23634 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:47:06.900285 23634 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:47:06.900565 23634 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:47:06.901437 23634 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 22:47:06.902580 23634 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:47:06.902695 23634 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:47:06.902796 23634 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:47:06.902858 23634 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 22:47:06.914245 23634 rpc_server.cc:307] RPC server started. Bound to: 127.10.198.3:37937
I20260925 22:47:06.918279 23634 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376259650642-11032-0/minicluster-data/ts-2/data/info.pb
I20260925 22:47:06.921162 11032 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrWXm8X/build/debug/bin/kudu as pid 23634
I20260925 22:47:06.921298 11032 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376259650642-11032-0/minicluster-data/ts-2/wal/instance
I20260925 22:47:06.921587 11032 external_mini_cluster.cc:1428] Setting key 744bdb7fd17d97b7c805fd8fc8aaa576
Sep 25 22:47:06 dist-test-slave-5vdr krb5kdc[22803](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376426, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.198.3@KRBTEST.COM for kudu/127.10.198.62@KRBTEST.COM
I20260925 22:47:06.920297 23762 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.198.3:37937 every 8 connection(s)
I20260925 22:47:06.943264 23496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:47:06.926246 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.198.3:49567 (local address 127.10.198.62:38359)
0925 22:47:06.926370 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0925 22:47:06.926375 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:47:06.926887 (+ 512us) server_negotiation.cc:407] Connection header received
0925 22:47:06.928198 (+ 1311us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:47:06.928201 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:47:06.928274 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:47:06.928325 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:47:06.930159 (+ 1834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:47:06.930962 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:47:06.932865 (+ 1903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:47:06.933240 (+ 375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:47:06.933330 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:47:06.937559 (+ 4229us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:47:06.937564 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:47:06.937566 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:47:06.937600 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:47:06.940454 (+ 2854us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:47:06.941239 (+ 785us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:47:06.941243 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:47:06.941245 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:47:06.941347 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:47:06.941841 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:47:06.941845 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:47:06.941847 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:47:06.942105 (+ 258us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:47:06.942193 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0925 22:47:06.942951 (+ 758us) server_negotiation.cc:299] Negotiation successful
0925 22:47:06.943093 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260925 22:47:06.945389 23765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:47:06.924501 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.198.62:38359 (local address 127.10.198.3:49567)
0925 22:47:06.926708 (+ 2207us) negotiation.cc:107] Waiting for socket to connect
0925 22:47:06.926774 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0925 22:47:06.927924 (+ 1150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:47:06.928767 (+ 843us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:47:06.928777 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:47:06.929168 (+ 391us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:47:06.929882 (+ 714us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:47:06.929976 (+ 94us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:47:06.931142 (+ 1166us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:47:06.931145 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:47:06.932691 (+ 1546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:47:06.932705 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:47:06.933578 (+ 873us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:47:06.933582 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:47:06.933818 (+ 236us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:47:06.934568 (+ 750us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:47:06.934592 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:47:06.937332 (+ 2740us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:47:06.940626 (+ 3294us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:47:06.940635 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:47:06.940658 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:47:06.941105 (+ 447us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:47:06.941467 (+ 362us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:47:06.941470 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:47:06.941472 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:47:06.941710 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:47:06.942266 (+ 556us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:47:06.942274 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:47:06.942642 (+ 368us) client_negotiation.cc:769] Sending connection context
0925 22:47:06.943811 (+ 1169us) client_negotiation.cc:240] Negotiation successful
0925 22:47:06.944252 (+ 441us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1978,"thread_start_us":667,"threads_started":1}
I20260925 22:47:06.947144 23763 heartbeater.cc:347] Connected to a master server at 127.10.198.62:38359
I20260925 22:47:06.947417 23763 heartbeater.cc:464] Registering TS with master...
I20260925 22:47:06.948067 23763 heartbeater.cc:511] Master 127.10.198.62:38359 requested a full tablet report, sending...
I20260925 22:47:06.949615 23279 ts_manager.cc:194] Registered new tserver with Master: 5196eab08c014dd48322cc60aa5b54bf (127.10.198.3:37937)
I20260925 22:47:06.950439 23279 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.198.3@KRBTEST.COM'} at 127.10.198.3:49567
I20260925 22:47:06.954648 11032 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 22:47:06.955341 11032 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrWXm8X/build/debug/bin/kudu with pid 23360
I20260925 22:47:06.963018 11032 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrWXm8X/build/debug/bin/kudu with pid 23499
I20260925 22:47:06.972587 11032 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrWXm8X/build/debug/bin/kudu with pid 23634
I20260925 22:47:06.981297 11032 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrWXm8X/build/debug/bin/kudu with pid 23247
I20260925 22:47:06.990670 11032 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrWXm8X/build/debug/bin/kudu
/tmp/dist-test-taskrWXm8X/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376259650642-11032-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376259650642-11032-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376259650642-11032-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376259650642-11032-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.198.62:38359
--webserver_interface=127.10.198.62
--webserver_port=40291
--keytab_file=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376259650642-11032-0/krb5kdc/kudu_127.10.198.62.keytab
--principal=kudu/127.10.198.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.198.20:34721
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.198.62:38359
--ranger_config_path=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376259650642-11032-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.10.198.20:47825/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.10.198.20:47825/kms with env {KRB5CCNAME=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376259650642-11032-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376259650642-11032-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376259650642-11032-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:47:07.109769 23766 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:47:07.110172 23766 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:47:07.110261 23766 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:47:07.114800 23766 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 22:47:07.114928 23766 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:47:07.114964 23766 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:47:07.114995 23766 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 22:47:07.115049 23766 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 22:47:07.121284 23766 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.198.20:34721
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376259650642-11032-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376259650642-11032-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.10.198.20:47825/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.10.198.62:38359
--ranger_config_path=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376259650642-11032-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376259650642-11032-0/krb5kdc/kudu_127.10.198.62.keytab
--principal=kudu/127.10.198.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.198.62:38359
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376259650642-11032-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.198.62
--webserver_port=40291
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23766
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376259650642-11032-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:09:16 UTC on ccycloud.quasar-afmhyw.root.comops.site
build id 4673379
I20260925 22:47:07.122541 23766 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:47:07.123687 23766 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 22:47:07.133293 23766 server_base.cc:1034] running on GCE node
W20260925 22:47:07.133235 23774 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:47:07.133235 23772 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:47:07.133235 23771 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:47:07.134298 23766 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:47:07.135637 23766 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:47:07.136849 23766 hybrid_clock.cc:648] HybridClock initialized: now 1790376427136828 us; error 50 us; skew 500 ppm
Sep 25 22:47:07 dist-test-slave-5vdr krb5kdc[22803](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376427, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.198.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:47:07.140547 23766 init.cc:377] Logged in from keytab as kudu/127.10.198.62@KRBTEST.COM (short username kudu)
I20260925 22:47:07.141857 23766 webserver.cc:460] Webserver started at http://127.10.198.62:40291/ using document root <none> and password file <none>
I20260925 22:47:07.142563 23766 fs_manager.cc:359] Metadata directory not provided
I20260925 22:47:07.142730 23766 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 25 22:47:07 dist-test-slave-5vdr krb5kdc[22803](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376427, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.198.62@KRBTEST.COM for HTTP/127.10.198.20@KRBTEST.COM
I20260925 22:47:07.210995 23766 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.004s sys 0.000s
I20260925 22:47:07.213753 23782 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:47:07.215420 23766 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260925 22:47:07.215636 23766 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376259650642-11032-0/minicluster-data/master-0/data,/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376259650642-11032-0/minicluster-data/master-0/wal
uuid: "ec9d2e7ae7f1418caff74bbf04252560"
format_stamp: "Formatted at 2026-09-25 22:47:04 on dist-test-slave-5vdr"
server_key: "a1975049dcac9369963f28f867b5fcba"
server_key_iv: "4f9767831b7e71bde5df4b5a3c46e8db"
server_key_version: "kuduclusterkey@0"
I20260925 22:47:07.216498 23766 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376259650642-11032-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376259650642-11032-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376259650642-11032-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:47:07.249107 23766 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:47:07.252576 23766 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:47:07.252820 23766 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:47:07.261360 23766 rpc_server.cc:307] RPC server started. Bound to: 127.10.198.62:38359
I20260925 22:47:07.261372 23834 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.198.62:38359 every 8 connection(s)
I20260925 22:47:07.262642 23766 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376259650642-11032-0/minicluster-data/master-0/data/info.pb
I20260925 22:47:07.267421 23835 sys_catalog.cc:263] Verifying existing consensus state
I20260925 22:47:07.269481 23835 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ec9d2e7ae7f1418caff74bbf04252560: Bootstrap starting.
I20260925 22:47:07.269984 11032 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrWXm8X/build/debug/bin/kudu as pid 23766
I20260925 22:47:07.270484 11032 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrWXm8X/build/debug/bin/kudu
/tmp/dist-test-taskrWXm8X/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376259650642-11032-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376259650642-11032-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376259650642-11032-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376259650642-11032-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.10.198.20:47825/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.198.1:36413
--local_ip_for_outbound_sockets=127.10.198.1
--tserver_master_addrs=127.10.198.62:38359
--webserver_port=39569
--webserver_interface=127.10.198.1
--keytab_file=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376259650642-11032-0/krb5kdc/kudu_127.10.198.1.keytab
--principal=kudu/127.10.198.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.198.20:34721
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.10.198.20:47825/kms with env {KRB5CCNAME=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376259650642-11032-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376259650642-11032-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376259650642-11032-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 22:47:07.287949 23835 log.cc:824] T 00000000000000000000000000000000 P ec9d2e7ae7f1418caff74bbf04252560: Log is configured to *not* fsync() on all Append() calls
I20260925 22:47:07.295625 23835 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ec9d2e7ae7f1418caff74bbf04252560: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 22:47:07.296129 23835 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ec9d2e7ae7f1418caff74bbf04252560: Bootstrap complete.
I20260925 22:47:07.300493 23835 raft_consensus.cc:348] T 00000000000000000000000000000000 P ec9d2e7ae7f1418caff74bbf04252560 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec9d2e7ae7f1418caff74bbf04252560" member_type: VOTER last_known_addr { host: "127.10.198.62" port: 38359 } }
I20260925 22:47:07.300972 23835 raft_consensus.cc:729] T 00000000000000000000000000000000 P ec9d2e7ae7f1418caff74bbf04252560 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: ec9d2e7ae7f1418caff74bbf04252560, State: Initialized, Role: FOLLOWER
I20260925 22:47:07.301573 23835 consensus_queue.cc:260] T 00000000000000000000000000000000 P ec9d2e7ae7f1418caff74bbf04252560 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec9d2e7ae7f1418caff74bbf04252560" member_type: VOTER last_known_addr { host: "127.10.198.62" port: 38359 } }
I20260925 22:47:07.301708 23835 raft_consensus.cc:388] T 00000000000000000000000000000000 P ec9d2e7ae7f1418caff74bbf04252560 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 22:47:07.301806 23835 raft_consensus.cc:482] T 00000000000000000000000000000000 P ec9d2e7ae7f1418caff74bbf04252560 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 22:47:07.301936 23835 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ec9d2e7ae7f1418caff74bbf04252560 [term 1 FOLLOWER]: Advancing to term 2
I20260925 22:47:07.304504 23835 raft_consensus.cc:504] T 00000000000000000000000000000000 P ec9d2e7ae7f1418caff74bbf04252560 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec9d2e7ae7f1418caff74bbf04252560" member_type: VOTER last_known_addr { host: "127.10.198.62" port: 38359 } }
I20260925 22:47:07.304935 23835 leader_election.cc:302] T 00000000000000000000000000000000 P ec9d2e7ae7f1418caff74bbf04252560 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ec9d2e7ae7f1418caff74bbf04252560; no voters:
I20260925 22:47:07.305377 23835 leader_election.cc:288] T 00000000000000000000000000000000 P ec9d2e7ae7f1418caff74bbf04252560 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260925 22:47:07.305477 23839 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ec9d2e7ae7f1418caff74bbf04252560 [term 2 FOLLOWER]: Leader election won for term 2
I20260925 22:47:07.305766 23839 raft_consensus.cc:686] T 00000000000000000000000000000000 P ec9d2e7ae7f1418caff74bbf04252560 [term 2 LEADER]: Becoming Leader. State: Replica: ec9d2e7ae7f1418caff74bbf04252560, State: Running, Role: LEADER
I20260925 22:47:07.306164 23839 consensus_queue.cc:237] T 00000000000000000000000000000000 P ec9d2e7ae7f1418caff74bbf04252560 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec9d2e7ae7f1418caff74bbf04252560" member_type: VOTER last_known_addr { host: "127.10.198.62" port: 38359 } }
I20260925 22:47:07.306769 23835 sys_catalog.cc:564] T 00000000000000000000000000000000 P ec9d2e7ae7f1418caff74bbf04252560 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 22:47:07.307751 23841 sys_catalog.cc:455] T 00000000000000000000000000000000 P ec9d2e7ae7f1418caff74bbf04252560 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ec9d2e7ae7f1418caff74bbf04252560. Latest consensus state: current_term: 2 leader_uuid: "ec9d2e7ae7f1418caff74bbf04252560" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec9d2e7ae7f1418caff74bbf04252560" member_type: VOTER last_known_addr { host: "127.10.198.62" port: 38359 } } }
I20260925 22:47:07.307925 23841 sys_catalog.cc:458] T 00000000000000000000000000000000 P ec9d2e7ae7f1418caff74bbf04252560 [sys.catalog]: This master's current role is: LEADER
I20260925 22:47:07.308200 23840 sys_catalog.cc:455] T 00000000000000000000000000000000 P ec9d2e7ae7f1418caff74bbf04252560 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "ec9d2e7ae7f1418caff74bbf04252560" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec9d2e7ae7f1418caff74bbf04252560" member_type: VOTER last_known_addr { host: "127.10.198.62" port: 38359 } } }
I20260925 22:47:07.308279 23840 sys_catalog.cc:458] T 00000000000000000000000000000000 P ec9d2e7ae7f1418caff74bbf04252560 [sys.catalog]: This master's current role is: LEADER
I20260925 22:47:07.308316 23848 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 22:47:07.312162 23848 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 22:47:07.312937 23835 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376259650642-11032-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260925 22:47:07.313010 23848 catalog_manager.cc:1259] Loaded cluster ID: 21f17c2efe554409a68eae6c72717ccc
I20260925 22:47:07.313067 23848 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 22:47:07.315958 23848 catalog_manager.cc:1504] Loading token signing keys...
I20260925 22:47:07.316862 23848 catalog_manager.cc:5879] T 00000000000000000000000000000000 P ec9d2e7ae7f1418caff74bbf04252560: Loaded TSK: 0
I20260925 22:47:07.318254 23848 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260925 22:47:07.408461 23837 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:47:07.409020 23837 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:47:07.409163 23837 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:47:07.414973 23837 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:47:07.415169 23837 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:47:07.415316 23837 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.198.1
I20260925 22:47:07.422634 23837 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.198.20:34721
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376259650642-11032-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376259650642-11032-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.10.198.20:47825/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376259650642-11032-0/krb5kdc/kudu_127.10.198.1.keytab
--principal=kudu/127.10.198.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.198.1:36413
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376259650642-11032-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.198.1
--webserver_port=39569
--tserver_master_addrs=127.10.198.62:38359
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23837
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.198.1
--log_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376259650642-11032-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:09:16 UTC on ccycloud.quasar-afmhyw.root.comops.site
build id 4673379
I20260925 22:47:07.424152 23837 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:47:07.425549 23837 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:47:07.437547 23873 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:47:07.437567 23875 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:47:07.438031 23837 server_base.cc:1034] running on GCE node
W20260925 22:47:07.437608 23872 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:47:07.439054 23837 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:47:07.440017 23837 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:47:07.443333 23837 hybrid_clock.cc:648] HybridClock initialized: now 1790376427443039 us; error 305 us; skew 500 ppm
Sep 25 22:47:07 dist-test-slave-5vdr krb5kdc[22803](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376427, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.198.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:47:07.448052 23837 init.cc:377] Logged in from keytab as kudu/127.10.198.1@KRBTEST.COM (short username kudu)
I20260925 22:47:07.449566 23837 webserver.cc:460] Webserver started at http://127.10.198.1:39569/ using document root <none> and password file <none>
I20260925 22:47:07.450461 23837 fs_manager.cc:359] Metadata directory not provided
I20260925 22:47:07.450598 23837 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 25 22:47:07 dist-test-slave-5vdr krb5kdc[22803](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376427, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.198.1@KRBTEST.COM for HTTP/127.10.198.20@KRBTEST.COM
I20260925 22:47:07.481809 23837 fs_manager.cc:709] Time spent opening directory manager: real 0.012s user 0.004s sys 0.000s
I20260925 22:47:07.488948 23883 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:47:07.491264 23837 fs_manager.cc:725] Time spent opening block manager: real 0.007s user 0.002s sys 0.000s
I20260925 22:47:07.491456 23837 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376259650642-11032-0/minicluster-data/ts-0/data,/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376259650642-11032-0/minicluster-data/ts-0/wal
uuid: "fea07c2152bc490580f2782c1319a37d"
format_stamp: "Formatted at 2026-09-25 22:47:06 on dist-test-slave-5vdr"
server_key: "69fda9a198d8db7f4346722c1ab2b01e"
server_key_iv: "bb662cddd2d49de55f0e9395f73a1741"
server_key_version: "kuduclusterkey@0"
I20260925 22:47:07.492480 23837 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376259650642-11032-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376259650642-11032-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376259650642-11032-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:47:07.511008 23837 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:47:07.520356 23837 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:47:07.520670 23837 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:47:07.521615 23837 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 22:47:07.522852 23837 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:47:07.522979 23837 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:47:07.523129 23837 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:47:07.523212 23837 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 22:47:07.535207 23837 rpc_server.cc:307] RPC server started. Bound to: 127.10.198.1:36413
I20260925 22:47:07.535264 23996 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.198.1:36413 every 8 connection(s)
I20260925 22:47:07.536952 23837 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376259650642-11032-0/minicluster-data/ts-0/data/info.pb
I20260925 22:47:07.544391 11032 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrWXm8X/build/debug/bin/kudu as pid 23837
I20260925 22:47:07.544914 11032 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrWXm8X/build/debug/bin/kudu
/tmp/dist-test-taskrWXm8X/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376259650642-11032-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376259650642-11032-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376259650642-11032-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376259650642-11032-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.10.198.20:47825/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.198.2:34763
--local_ip_for_outbound_sockets=127.10.198.2
--tserver_master_addrs=127.10.198.62:38359
--webserver_port=35411
--webserver_interface=127.10.198.2
--keytab_file=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376259650642-11032-0/krb5kdc/kudu_127.10.198.2.keytab
--principal=kudu/127.10.198.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.198.20:34721
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.10.198.20:47825/kms with env {KRB5CCNAME=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376259650642-11032-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376259650642-11032-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376259650642-11032-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 25 22:47:07 dist-test-slave-5vdr krb5kdc[22803](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376427, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.198.1@KRBTEST.COM for kudu/127.10.198.62@KRBTEST.COM
I20260925 22:47:07.564311 24000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:47:07.541881 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.198.1:52385 (local address 127.10.198.62:38359)
0925 22:47:07.542520 (+ 639us) server_negotiation.cc:206] Beginning negotiation
0925 22:47:07.542544 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0925 22:47:07.543120 (+ 576us) server_negotiation.cc:407] Connection header received
0925 22:47:07.544854 (+ 1734us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:47:07.544872 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:47:07.545340 (+ 468us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:47:07.545731 (+ 391us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:47:07.548242 (+ 2511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:47:07.549878 (+ 1636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:47:07.551692 (+ 1814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:47:07.552518 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:47:07.552688 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:47:07.557332 (+ 4644us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:47:07.557339 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:47:07.557356 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:47:07.557392 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:47:07.559342 (+ 1950us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:47:07.560161 (+ 819us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:47:07.560170 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:47:07.560176 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:47:07.560399 (+ 223us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:47:07.560969 (+ 570us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:47:07.560974 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:47:07.560975 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:47:07.561576 (+ 601us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:47:07.561718 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0925 22:47:07.562650 (+ 932us) server_negotiation.cc:299] Negotiation successful
0925 22:47:07.563051 (+ 401us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":471,"thread_start_us":110,"threads_started":1}
I20260925 22:47:07.566998 23999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:47:07.541630 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.198.62:38359 (local address 127.10.198.1:52385)
0925 22:47:07.542868 (+ 1238us) negotiation.cc:107] Waiting for socket to connect
0925 22:47:07.542933 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0925 22:47:07.544119 (+ 1186us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:47:07.545832 (+ 1713us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:47:07.545842 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:47:07.546207 (+ 365us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:47:07.547990 (+ 1783us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:47:07.548019 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:47:07.550112 (+ 2093us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:47:07.550116 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:47:07.551464 (+ 1348us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:47:07.551478 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:47:07.553018 (+ 1540us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:47:07.553024 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:47:07.553213 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:47:07.554922 (+ 1709us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:47:07.554942 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:47:07.556929 (+ 1987us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:47:07.559501 (+ 2572us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:47:07.559509 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:47:07.559526 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:47:07.559982 (+ 456us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:47:07.560545 (+ 563us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:47:07.560548 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:47:07.560551 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:47:07.560811 (+ 260us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:47:07.561853 (+ 1042us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:47:07.561860 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:47:07.562235 (+ 375us) client_negotiation.cc:769] Sending connection context
0925 22:47:07.565216 (+ 2981us) client_negotiation.cc:240] Negotiation successful
0925 22:47:07.565693 (+ 477us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":987,"thread_start_us":178,"threads_started":1}
I20260925 22:47:07.571130 23997 heartbeater.cc:347] Connected to a master server at 127.10.198.62:38359
I20260925 22:47:07.571477 23997 heartbeater.cc:464] Registering TS with master...
I20260925 22:47:07.572412 23997 heartbeater.cc:511] Master 127.10.198.62:38359 requested a full tablet report, sending...
W20260925 22:47:07.575479 23997 heartbeater.cc:644] Failed to heartbeat to 127.10.198.62:38359 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260925 22:47:07.575659 23997 heartbeater.cc:464] Registering TS with master...
I20260925 22:47:07.576107 23997 heartbeater.cc:511] Master 127.10.198.62:38359 requested a full tablet report, sending...
I20260925 22:47:07.577392 23997 heartbeater.cc:464] Registering TS with master...
I20260925 22:47:07.577631 23997 heartbeater.cc:511] Master 127.10.198.62:38359 requested a full tablet report, sending...
W20260925 22:47:07.578653 23997 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260925 22:47:07.704897 24001 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:47:07.705415 24001 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:47:07.705543 24001 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:47:07.711879 24001 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:47:07.712061 24001 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:47:07.712200 24001 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.198.2
I20260925 22:47:07.718746 24001 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.198.20:34721
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376259650642-11032-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376259650642-11032-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.10.198.20:47825/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376259650642-11032-0/krb5kdc/kudu_127.10.198.2.keytab
--principal=kudu/127.10.198.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.198.2:34763
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376259650642-11032-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.198.2
--webserver_port=35411
--tserver_master_addrs=127.10.198.62:38359
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24001
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.198.2
--log_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376259650642-11032-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:09:16 UTC on ccycloud.quasar-afmhyw.root.comops.site
build id 4673379
I20260925 22:47:07.720119 24001 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:47:07.721469 24001 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:47:07.733908 24009 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:47:07.734406 24007 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:47:07.734524 24006 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:47:07.735805 24001 server_base.cc:1034] running on GCE node
I20260925 22:47:07.737027 24001 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:47:07.737983 24001 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:47:07.741496 24001 hybrid_clock.cc:648] HybridClock initialized: now 1790376427741439 us; error 93 us; skew 500 ppm
Sep 25 22:47:07 dist-test-slave-5vdr krb5kdc[22803](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376427, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.198.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:47:07.746037 24001 init.cc:377] Logged in from keytab as kudu/127.10.198.2@KRBTEST.COM (short username kudu)
I20260925 22:47:07.747509 24001 webserver.cc:460] Webserver started at http://127.10.198.2:35411/ using document root <none> and password file <none>
I20260925 22:47:07.748435 24001 fs_manager.cc:359] Metadata directory not provided
I20260925 22:47:07.748579 24001 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 25 22:47:07 dist-test-slave-5vdr krb5kdc[22803](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376427, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.198.2@KRBTEST.COM for HTTP/127.10.198.20@KRBTEST.COM
I20260925 22:47:07.772012 24001 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.005s sys 0.000s
I20260925 22:47:07.777022 24017 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:47:07.779235 24001 fs_manager.cc:725] Time spent opening block manager: real 0.006s user 0.003s sys 0.000s
I20260925 22:47:07.779430 24001 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376259650642-11032-0/minicluster-data/ts-1/data,/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376259650642-11032-0/minicluster-data/ts-1/wal
uuid: "01b6959223424811be44029b72bdac12"
format_stamp: "Formatted at 2026-09-25 22:47:06 on dist-test-slave-5vdr"
server_key: "647135544f5962dd2b6a6aa272c19ecb"
server_key_iv: "a36a0d6841e64ab5b2426fcfa2030bbc"
server_key_version: "kuduclusterkey@0"
I20260925 22:47:07.780575 24001 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376259650642-11032-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376259650642-11032-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376259650642-11032-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:47:07.811062 24001 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:47:07.814548 24001 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:47:07.814796 24001 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:47:07.815507 24001 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 22:47:07.816594 24001 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:47:07.816728 24001 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:47:07.816808 24001 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:47:07.816864 24001 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 22:47:07.830760 24001 rpc_server.cc:307] RPC server started. Bound to: 127.10.198.2:34763
I20260925 22:47:07.832408 24001 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376259650642-11032-0/minicluster-data/ts-1/data/info.pb
I20260925 22:47:07.834903 24130 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.198.2:34763 every 8 connection(s)
I20260925 22:47:07.841535 11032 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrWXm8X/build/debug/bin/kudu as pid 24001
I20260925 22:47:07.842101 11032 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrWXm8X/build/debug/bin/kudu
/tmp/dist-test-taskrWXm8X/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376259650642-11032-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376259650642-11032-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376259650642-11032-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376259650642-11032-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.10.198.20:47825/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.198.3:37937
--local_ip_for_outbound_sockets=127.10.198.3
--tserver_master_addrs=127.10.198.62:38359
--webserver_port=39873
--webserver_interface=127.10.198.3
--keytab_file=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376259650642-11032-0/krb5kdc/kudu_127.10.198.3.keytab
--principal=kudu/127.10.198.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.198.20:34721
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.10.198.20:47825/kms with env {KRB5CCNAME=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376259650642-11032-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376259650642-11032-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376259650642-11032-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 25 22:47:07 dist-test-slave-5vdr krb5kdc[22803](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376427, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.198.2@KRBTEST.COM for kudu/127.10.198.62@KRBTEST.COM
I20260925 22:47:07.851667 24000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:47:07.836602 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.198.2:37691 (local address 127.10.198.62:38359)
0925 22:47:07.836733 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0925 22:47:07.836740 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 22:47:07.836759 (+ 19us) server_negotiation.cc:407] Connection header received
0925 22:47:07.837430 (+ 671us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:47:07.837434 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:47:07.837506 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:47:07.837602 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:47:07.839529 (+ 1927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:47:07.840422 (+ 893us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:47:07.842683 (+ 2261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:47:07.842959 (+ 276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:47:07.843098 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:47:07.847095 (+ 3997us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:47:07.847099 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:47:07.847101 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:47:07.847133 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:47:07.848874 (+ 1741us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:47:07.849598 (+ 724us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:47:07.849602 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:47:07.849604 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:47:07.849704 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:47:07.850260 (+ 556us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:47:07.850264 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:47:07.850266 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:47:07.850499 (+ 233us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:47:07.850621 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0925 22:47:07.851320 (+ 699us) server_negotiation.cc:299] Negotiation successful
0925 22:47:07.851489 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260925 22:47:07.853660 24133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:47:07.835362 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.198.62:38359 (local address 127.10.198.2:37691)
0925 22:47:07.835974 (+ 612us) negotiation.cc:107] Waiting for socket to connect
0925 22:47:07.836032 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0925 22:47:07.837116 (+ 1084us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:47:07.837953 (+ 837us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:47:07.837964 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:47:07.838382 (+ 418us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:47:07.839230 (+ 848us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:47:07.839255 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:47:07.840632 (+ 1377us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:47:07.840635 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:47:07.842499 (+ 1864us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:47:07.842517 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:47:07.843157 (+ 640us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:47:07.843161 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:47:07.843369 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:47:07.845149 (+ 1780us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:47:07.845170 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:47:07.846781 (+ 1611us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:47:07.849051 (+ 2270us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:47:07.849059 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:47:07.849083 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:47:07.849466 (+ 383us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:47:07.849854 (+ 388us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:47:07.849858 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:47:07.849860 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:47:07.850137 (+ 277us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:47:07.850623 (+ 486us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:47:07.850634 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:47:07.850987 (+ 353us) client_negotiation.cc:769] Sending connection context
0925 22:47:07.851981 (+ 994us) client_negotiation.cc:240] Negotiation successful
0925 22:47:07.852432 (+ 451us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":337,"thread_start_us":200,"threads_started":1}
I20260925 22:47:07.855110 24131 heartbeater.cc:347] Connected to a master server at 127.10.198.62:38359
I20260925 22:47:07.855403 24131 heartbeater.cc:464] Registering TS with master...
I20260925 22:47:07.856065 24131 heartbeater.cc:511] Master 127.10.198.62:38359 requested a full tablet report, sending...
W20260925 22:47:07.858094 24131 heartbeater.cc:644] Failed to heartbeat to 127.10.198.62:38359 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260925 22:47:07.858282 24131 heartbeater.cc:464] Registering TS with master...
I20260925 22:47:07.858521 24131 heartbeater.cc:511] Master 127.10.198.62:38359 requested a full tablet report, sending...
I20260925 22:47:07.859586 24131 heartbeater.cc:464] Registering TS with master...
I20260925 22:47:07.859830 24131 heartbeater.cc:511] Master 127.10.198.62:38359 requested a full tablet report, sending...
W20260925 22:47:07.860823 24131 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260925 22:47:08.026737 24134 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:47:08.027209 24134 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:47:08.027295 24134 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:47:08.034265 24134 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:47:08.034370 24134 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:47:08.034473 24134 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.198.3
I20260925 22:47:08.043735 24134 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.198.20:34721
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376259650642-11032-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376259650642-11032-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.10.198.20:47825/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376259650642-11032-0/krb5kdc/kudu_127.10.198.3.keytab
--principal=kudu/127.10.198.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.198.3:37937
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376259650642-11032-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.198.3
--webserver_port=39873
--tserver_master_addrs=127.10.198.62:38359
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24134
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.198.3
--log_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376259650642-11032-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:09:16 UTC on ccycloud.quasar-afmhyw.root.comops.site
build id 4673379
I20260925 22:47:08.045547 24134 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:47:08.047220 24134 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:47:08.061187 24143 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:47:08.061403 24140 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:47:08.061203 24141 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:47:08.062105 24134 server_base.cc:1034] running on GCE node
I20260925 22:47:08.062613 24134 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:47:08.063369 24134 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:47:08.064605 24134 hybrid_clock.cc:648] HybridClock initialized: now 1790376428064566 us; error 61 us; skew 500 ppm
Sep 25 22:47:08 dist-test-slave-5vdr krb5kdc[22803](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376428, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.198.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:47:08.068259 24134 init.cc:377] Logged in from keytab as kudu/127.10.198.3@KRBTEST.COM (short username kudu)
I20260925 22:47:08.069630 24134 webserver.cc:460] Webserver started at http://127.10.198.3:39873/ using document root <none> and password file <none>
I20260925 22:47:08.070463 24134 fs_manager.cc:359] Metadata directory not provided
I20260925 22:47:08.070580 24134 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 25 22:47:08 dist-test-slave-5vdr krb5kdc[22803](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376428, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.198.3@KRBTEST.COM for HTTP/127.10.198.20@KRBTEST.COM
I20260925 22:47:08.106890 24134 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.004s sys 0.000s
I20260925 22:47:08.110803 24151 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:47:08.113210 24134 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.002s sys 0.000s
I20260925 22:47:08.113387 24134 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376259650642-11032-0/minicluster-data/ts-2/data,/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376259650642-11032-0/minicluster-data/ts-2/wal
uuid: "5196eab08c014dd48322cc60aa5b54bf"
format_stamp: "Formatted at 2026-09-25 22:47:06 on dist-test-slave-5vdr"
server_key: "5e61f155fb57bd9de22fd7a5e2808f5c"
server_key_iv: "94a4bcdac39dc112f9180932adf16430"
server_key_version: "kuduclusterkey@0"
I20260925 22:47:08.114401 24134 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376259650642-11032-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376259650642-11032-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376259650642-11032-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:47:08.130461 24134 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:47:08.134286 24134 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:47:08.134578 24134 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:47:08.135427 24134 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 22:47:08.136615 24134 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:47:08.136699 24134 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:47:08.136780 24134 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:47:08.136838 24134 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 22:47:08.148418 24134 rpc_server.cc:307] RPC server started. Bound to: 127.10.198.3:37937
I20260925 22:47:08.148597 24264 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.198.3:37937 every 8 connection(s)
I20260925 22:47:08.149672 24134 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376259650642-11032-0/minicluster-data/ts-2/data/info.pb
I20260925 22:47:08.153302 11032 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrWXm8X/build/debug/bin/kudu as pid 24134
Sep 25 22:47:08 dist-test-slave-5vdr krb5kdc[22803](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376428, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.198.3@KRBTEST.COM for kudu/127.10.198.62@KRBTEST.COM
I20260925 22:47:08.170764 24000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:47:08.152131 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.198.3:35807 (local address 127.10.198.62:38359)
0925 22:47:08.152282 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0925 22:47:08.152290 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 22:47:08.153649 (+ 1359us) server_negotiation.cc:407] Connection header received
0925 22:47:08.155703 (+ 2054us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:47:08.155707 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:47:08.155763 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:47:08.155872 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:47:08.157501 (+ 1629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:47:08.158305 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:47:08.160363 (+ 2058us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:47:08.160728 (+ 365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:47:08.160886 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:47:08.164859 (+ 3973us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:47:08.164864 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:47:08.164867 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:47:08.164909 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:47:08.167028 (+ 2119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:47:08.167953 (+ 925us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:47:08.167959 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:47:08.167960 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:47:08.168085 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:47:08.168690 (+ 605us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:47:08.168695 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:47:08.168699 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:47:08.168962 (+ 263us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:47:08.169056 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0925 22:47:08.170399 (+ 1343us) server_negotiation.cc:299] Negotiation successful
0925 22:47:08.170575 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260925 22:47:08.171866 24267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:47:08.152752 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.198.62:38359 (local address 127.10.198.3:35807)
0925 22:47:08.153437 (+ 685us) negotiation.cc:107] Waiting for socket to connect
0925 22:47:08.153512 (+ 75us) client_negotiation.cc:174] Beginning negotiation
0925 22:47:08.155396 (+ 1884us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:47:08.156108 (+ 712us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:47:08.156121 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:47:08.156498 (+ 377us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:47:08.157293 (+ 795us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:47:08.157319 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:47:08.158507 (+ 1188us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:47:08.158510 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:47:08.160176 (+ 1666us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:47:08.160195 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:47:08.161151 (+ 956us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:47:08.161156 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:47:08.161420 (+ 264us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:47:08.162451 (+ 1031us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:47:08.162475 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:47:08.164544 (+ 2069us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:47:08.167238 (+ 2694us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:47:08.167246 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:47:08.167266 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:47:08.167750 (+ 484us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:47:08.168237 (+ 487us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:47:08.168241 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:47:08.168245 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:47:08.168540 (+ 295us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:47:08.169563 (+ 1023us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:47:08.169572 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:47:08.170007 (+ 435us) client_negotiation.cc:769] Sending connection context
0925 22:47:08.170261 (+ 254us) client_negotiation.cc:240] Negotiation successful
0925 22:47:08.170678 (+ 417us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":384,"thread_start_us":193,"threads_started":1}
I20260925 22:47:08.173403 24265 heartbeater.cc:347] Connected to a master server at 127.10.198.62:38359
I20260925 22:47:08.173695 24265 heartbeater.cc:464] Registering TS with master...
I20260925 22:47:08.174352 24265 heartbeater.cc:511] Master 127.10.198.62:38359 requested a full tablet report, sending...
W20260925 22:47:08.176383 24265 heartbeater.cc:644] Failed to heartbeat to 127.10.198.62:38359 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260925 22:47:08.176585 24265 heartbeater.cc:464] Registering TS with master...
I20260925 22:47:08.176805 24265 heartbeater.cc:511] Master 127.10.198.62:38359 requested a full tablet report, sending...
I20260925 22:47:08.178264 24265 heartbeater.cc:464] Registering TS with master...
I20260925 22:47:08.178543 24265 heartbeater.cc:511] Master 127.10.198.62:38359 requested a full tablet report, sending...
W20260925 22:47:08.179880 24265 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260925 22:47:08.181573 24269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:47:08.154054 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60730 (local address 127.10.198.62:38359)
0925 22:47:08.154680 (+ 626us) server_negotiation.cc:206] Beginning negotiation
0925 22:47:08.154690 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 22:47:08.154705 (+ 15us) server_negotiation.cc:407] Connection header received
0925 22:47:08.154749 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:47:08.154752 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:47:08.154801 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:47:08.154903 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:47:08.155987 (+ 1084us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:47:08.156827 (+ 840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:47:08.164772 (+ 7945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:47:08.165194 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:47:08.165715 (+ 521us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:47:08.169129 (+ 3414us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:47:08.169139 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:47:08.169142 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:47:08.169161 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:47:08.178924 (+ 9763us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:47:08.179602 (+ 678us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:47:08.179606 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:47:08.179608 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:47:08.179716 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:47:08.180379 (+ 663us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:47:08.180384 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:47:08.180385 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:47:08.180670 (+ 285us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:47:08.180961 (+ 291us) server_negotiation.cc:1035] Waiting for connection context
0925 22:47:08.181205 (+ 244us) server_negotiation.cc:299] Negotiation successful
0925 22:47:08.181374 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":546,"thread_start_us":95,"threads_started":1}
I20260925 22:47:08.579038 23997 heartbeater.cc:464] Registering TS with master...
I20260925 22:47:08.579391 23997 heartbeater.cc:511] Master 127.10.198.62:38359 requested a full tablet report, sending...
I20260925 22:47:08.580605 23799 ts_manager.cc:194] Registered new tserver with Master: fea07c2152bc490580f2782c1319a37d (127.10.198.1:36413)
I20260925 22:47:08.581899 23799 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.198.1@KRBTEST.COM'} at 127.10.198.1:52385
Sep 25 22:47:08 dist-test-slave-5vdr krb5kdc[22803](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790376428, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.198.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:47:08.861092 24131 heartbeater.cc:464] Registering TS with master...
I20260925 22:47:08.861416 24131 heartbeater.cc:511] Master 127.10.198.62:38359 requested a full tablet report, sending...
I20260925 22:47:08.862303 23799 ts_manager.cc:194] Registered new tserver with Master: 01b6959223424811be44029b72bdac12 (127.10.198.2:34763)
I20260925 22:47:08.862957 23799 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.198.2@KRBTEST.COM'} at 127.10.198.2:37691
Sep 25 22:47:09 dist-test-slave-5vdr krb5kdc[22803](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376428, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.198.62@KRBTEST.COM for HTTP/127.10.198.20@KRBTEST.COM
I20260925 22:47:09.180202 24265 heartbeater.cc:464] Registering TS with master...
I20260925 22:47:09.180611 24265 heartbeater.cc:511] Master 127.10.198.62:38359 requested a full tablet report, sending...
I20260925 22:47:09.181676 23799 ts_manager.cc:194] Registered new tserver with Master: 5196eab08c014dd48322cc60aa5b54bf (127.10.198.3:37937)
I20260925 22:47:09.182511 23799 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.198.3@KRBTEST.COM'} at 127.10.198.3:35807
I20260925 22:47:09.189738 11032 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 22:47:09.202046 24287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:47:09.191606 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60738 (local address 127.10.198.62:38359)
0925 22:47:09.191945 (+ 339us) server_negotiation.cc:206] Beginning negotiation
0925 22:47:09.191950 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:47:09.191967 (+ 17us) server_negotiation.cc:407] Connection header received
0925 22:47:09.192064 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:47:09.192068 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:47:09.192117 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:47:09.192212 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:47:09.193179 (+ 967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:47:09.194037 (+ 858us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:47:09.195382 (+ 1345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:47:09.195778 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:47:09.195946 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:47:09.197789 (+ 1843us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:47:09.197794 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:47:09.197797 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:47:09.197834 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:47:09.199891 (+ 2057us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:47:09.200472 (+ 581us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:47:09.200476 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:47:09.200478 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:47:09.200577 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:47:09.200889 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:47:09.200892 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:47:09.200894 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:47:09.201135 (+ 241us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:47:09.201265 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0925 22:47:09.201534 (+ 269us) server_negotiation.cc:299] Negotiation successful
0925 22:47:09.201675 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":228,"thread_start_us":85,"threads_started":1}
I20260925 22:47:09.206820 23799 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60738:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 22:47:09.211380 23799 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 22:47:09.235147 24292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:47:09.223370 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.198.2:34763 (local address 127.0.0.1:56924)
0925 22:47:09.228350 (+ 4980us) negotiation.cc:107] Waiting for socket to connect
0925 22:47:09.228415 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0925 22:47:09.228552 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:47:09.229575 (+ 1023us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:47:09.229582 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:47:09.229606 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:47:09.229781 (+ 175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:47:09.229789 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:47:09.231499 (+ 1710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:47:09.231503 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:47:09.233036 (+ 1533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:47:09.233048 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:47:09.234522 (+ 1474us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:47:09.234526 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:47:09.234653 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:47:09.234697 (+ 44us) client_negotiation.cc:769] Sending connection context
0925 22:47:09.234819 (+ 122us) client_negotiation.cc:240] Negotiation successful
0925 22:47:09.234976 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4763,"thread_start_us":146,"threads_started":1}
I20260925 22:47:09.236193 24293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:47:09.223603 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56924 (local address 127.10.198.2:34763)
0925 22:47:09.229075 (+ 5472us) server_negotiation.cc:206] Beginning negotiation
0925 22:47:09.229082 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 22:47:09.229107 (+ 25us) server_negotiation.cc:407] Connection header received
0925 22:47:09.229179 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:47:09.229183 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:47:09.229393 (+ 210us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:47:09.229580 (+ 187us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:47:09.229937 (+ 357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:47:09.231338 (+ 1401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:47:09.233207 (+ 1869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:47:09.234138 (+ 931us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:47:09.234262 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:47:09.234448 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0925 22:47:09.235902 (+ 1454us) server_negotiation.cc:299] Negotiation successful
0925 22:47:09.236037 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5368,"thread_start_us":93,"threads_started":1}
I20260925 22:47:09.237587 24294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:47:09.225529 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.198.3:37937 (local address 127.0.0.1:41826)
0925 22:47:09.230090 (+ 4561us) negotiation.cc:107] Waiting for socket to connect
0925 22:47:09.230104 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 22:47:09.230207 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:47:09.231957 (+ 1750us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:47:09.231961 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:47:09.231980 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:47:09.232067 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:47:09.232073 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:47:09.233420 (+ 1347us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:47:09.233424 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:47:09.235667 (+ 2243us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:47:09.235680 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:47:09.237132 (+ 1452us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:47:09.237136 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:47:09.237237 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:47:09.237264 (+ 27us) client_negotiation.cc:769] Sending connection context
0925 22:47:09.237345 (+ 81us) client_negotiation.cc:240] Negotiation successful
0925 22:47:09.237435 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4502,"thread_start_us":97,"threads_started":1}
I20260925 22:47:09.238431 24297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:47:09.225935 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41826 (local address 127.10.198.3:37937)
0925 22:47:09.229946 (+ 4011us) server_negotiation.cc:206] Beginning negotiation
0925 22:47:09.229953 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 22:47:09.230310 (+ 357us) server_negotiation.cc:407] Connection header received
0925 22:47:09.231285 (+ 975us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:47:09.231294 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:47:09.231583 (+ 289us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:47:09.231743 (+ 160us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:47:09.232213 (+ 470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:47:09.233293 (+ 1080us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:47:09.235829 (+ 2536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:47:09.236767 (+ 938us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:47:09.236897 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:47:09.237056 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0925 22:47:09.238157 (+ 1101us) server_negotiation.cc:299] Negotiation successful
0925 22:47:09.238285 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3869,"thread_start_us":86,"threads_started":1}
I20260925 22:47:09.240038 24065 tablet_service.cc:1467] Processing CreateTablet for tablet 0149d246c0b44093a47d45c813a47d69 (DEFAULT_TABLE table=test-table [id=559b38368de9403db2c6a0dc6aa1b6fc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 22:47:09.241573 24065 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0149d246c0b44093a47d45c813a47d69. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:47:09.242195 24199 tablet_service.cc:1467] Processing CreateTablet for tablet 0149d246c0b44093a47d45c813a47d69 (DEFAULT_TABLE table=test-table [id=559b38368de9403db2c6a0dc6aa1b6fc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 22:47:09.243685 24199 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0149d246c0b44093a47d45c813a47d69. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:47:09.248274 24295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:47:09.225322 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.198.1:36413 (local address 127.0.0.1:54956)
0925 22:47:09.228777 (+ 3455us) negotiation.cc:107] Waiting for socket to connect
0925 22:47:09.228788 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 22:47:09.228847 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:47:09.241555 (+ 12708us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:47:09.241559 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:47:09.241585 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:47:09.241738 (+ 153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:47:09.241746 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:47:09.243950 (+ 2204us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:47:09.243956 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:47:09.245153 (+ 1197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:47:09.245170 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:47:09.247712 (+ 2542us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:47:09.247717 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:47:09.247831 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:47:09.247904 (+ 73us) client_negotiation.cc:769] Sending connection context
0925 22:47:09.247963 (+ 59us) client_negotiation.cc:240] Negotiation successful
0925 22:47:09.248068 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3401,"thread_start_us":124,"threads_started":1}
I20260925 22:47:09.249424 24296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:47:09.225852 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54956 (local address 127.10.198.1:36413)
0925 22:47:09.241004 (+ 15152us) server_negotiation.cc:206] Beginning negotiation
0925 22:47:09.241012 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 22:47:09.241042 (+ 30us) server_negotiation.cc:407] Connection header received
0925 22:47:09.241129 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:47:09.241135 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:47:09.241363 (+ 228us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:47:09.241935 (+ 572us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:47:09.242111 (+ 176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:47:09.243696 (+ 1585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:47:09.245380 (+ 1684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:47:09.247532 (+ 2152us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:47:09.248849 (+ 1317us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:47:09.249067 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0925 22:47:09.249138 (+ 71us) server_negotiation.cc:299] Negotiation successful
0925 22:47:09.249259 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14977,"thread_start_us":97,"threads_started":1}
I20260925 22:47:09.253150 23919 tablet_service.cc:1467] Processing CreateTablet for tablet 0149d246c0b44093a47d45c813a47d69 (DEFAULT_TABLE table=test-table [id=559b38368de9403db2c6a0dc6aa1b6fc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 22:47:09.254098 24299 tablet_bootstrap.cc:492] T 0149d246c0b44093a47d45c813a47d69 P 5196eab08c014dd48322cc60aa5b54bf: Bootstrap starting.
I20260925 22:47:09.254619 23919 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0149d246c0b44093a47d45c813a47d69. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:47:09.256752 24299 tablet_bootstrap.cc:654] T 0149d246c0b44093a47d45c813a47d69 P 5196eab08c014dd48322cc60aa5b54bf: Neither blocks nor log segments found. Creating new log.
I20260925 22:47:09.257759 24299 log.cc:824] T 0149d246c0b44093a47d45c813a47d69 P 5196eab08c014dd48322cc60aa5b54bf: Log is configured to *not* fsync() on all Append() calls
I20260925 22:47:09.260365 24298 tablet_bootstrap.cc:492] T 0149d246c0b44093a47d45c813a47d69 P 01b6959223424811be44029b72bdac12: Bootstrap starting.
I20260925 22:47:09.263146 24298 tablet_bootstrap.cc:654] T 0149d246c0b44093a47d45c813a47d69 P 01b6959223424811be44029b72bdac12: Neither blocks nor log segments found. Creating new log.
I20260925 22:47:09.264356 24298 log.cc:824] T 0149d246c0b44093a47d45c813a47d69 P 01b6959223424811be44029b72bdac12: Log is configured to *not* fsync() on all Append() calls
I20260925 22:47:09.273959 24302 tablet_bootstrap.cc:492] T 0149d246c0b44093a47d45c813a47d69 P fea07c2152bc490580f2782c1319a37d: Bootstrap starting.
I20260925 22:47:09.274353 24299 tablet_bootstrap.cc:492] T 0149d246c0b44093a47d45c813a47d69 P 5196eab08c014dd48322cc60aa5b54bf: No bootstrap required, opened a new log
I20260925 22:47:09.274595 24299 ts_tablet_manager.cc:1397] T 0149d246c0b44093a47d45c813a47d69 P 5196eab08c014dd48322cc60aa5b54bf: Time spent bootstrapping tablet: real 0.021s user 0.004s sys 0.000s
I20260925 22:47:09.277105 24302 tablet_bootstrap.cc:654] T 0149d246c0b44093a47d45c813a47d69 P fea07c2152bc490580f2782c1319a37d: Neither blocks nor log segments found. Creating new log.
I20260925 22:47:09.278266 24302 log.cc:824] T 0149d246c0b44093a47d45c813a47d69 P fea07c2152bc490580f2782c1319a37d: Log is configured to *not* fsync() on all Append() calls
I20260925 22:47:09.279126 24299 raft_consensus.cc:348] T 0149d246c0b44093a47d45c813a47d69 P 5196eab08c014dd48322cc60aa5b54bf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01b6959223424811be44029b72bdac12" member_type: VOTER last_known_addr { host: "127.10.198.2" port: 34763 } } peers { permanent_uuid: "fea07c2152bc490580f2782c1319a37d" member_type: VOTER last_known_addr { host: "127.10.198.1" port: 36413 } } peers { permanent_uuid: "5196eab08c014dd48322cc60aa5b54bf" member_type: VOTER last_known_addr { host: "127.10.198.3" port: 37937 } }
I20260925 22:47:09.279484 24299 raft_consensus.cc:374] T 0149d246c0b44093a47d45c813a47d69 P 5196eab08c014dd48322cc60aa5b54bf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:47:09.279542 24299 raft_consensus.cc:729] T 0149d246c0b44093a47d45c813a47d69 P 5196eab08c014dd48322cc60aa5b54bf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5196eab08c014dd48322cc60aa5b54bf, State: Initialized, Role: FOLLOWER
I20260925 22:47:09.280328 24299 consensus_queue.cc:260] T 0149d246c0b44093a47d45c813a47d69 P 5196eab08c014dd48322cc60aa5b54bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01b6959223424811be44029b72bdac12" member_type: VOTER last_known_addr { host: "127.10.198.2" port: 34763 } } peers { permanent_uuid: "fea07c2152bc490580f2782c1319a37d" member_type: VOTER last_known_addr { host: "127.10.198.1" port: 36413 } } peers { permanent_uuid: "5196eab08c014dd48322cc60aa5b54bf" member_type: VOTER last_known_addr { host: "127.10.198.3" port: 37937 } }
I20260925 22:47:09.281544 24302 tablet_bootstrap.cc:492] T 0149d246c0b44093a47d45c813a47d69 P fea07c2152bc490580f2782c1319a37d: No bootstrap required, opened a new log
I20260925 22:47:09.281641 24299 ts_tablet_manager.cc:1428] T 0149d246c0b44093a47d45c813a47d69 P 5196eab08c014dd48322cc60aa5b54bf: Time spent starting tablet: real 0.007s user 0.004s sys 0.004s
I20260925 22:47:09.281778 24302 ts_tablet_manager.cc:1397] T 0149d246c0b44093a47d45c813a47d69 P fea07c2152bc490580f2782c1319a37d: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260925 22:47:09.284189 24265 heartbeater.cc:503] Master 127.10.198.62:38359 was elected leader, sending a full tablet report...
I20260925 22:47:09.284555 24298 tablet_bootstrap.cc:492] T 0149d246c0b44093a47d45c813a47d69 P 01b6959223424811be44029b72bdac12: No bootstrap required, opened a new log
I20260925 22:47:09.284775 24298 ts_tablet_manager.cc:1397] T 0149d246c0b44093a47d45c813a47d69 P 01b6959223424811be44029b72bdac12: Time spent bootstrapping tablet: real 0.025s user 0.006s sys 0.000s
I20260925 22:47:09.289274 24298 raft_consensus.cc:348] T 0149d246c0b44093a47d45c813a47d69 P 01b6959223424811be44029b72bdac12 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01b6959223424811be44029b72bdac12" member_type: VOTER last_known_addr { host: "127.10.198.2" port: 34763 } } peers { permanent_uuid: "fea07c2152bc490580f2782c1319a37d" member_type: VOTER last_known_addr { host: "127.10.198.1" port: 36413 } } peers { permanent_uuid: "5196eab08c014dd48322cc60aa5b54bf" member_type: VOTER last_known_addr { host: "127.10.198.3" port: 37937 } }
I20260925 22:47:09.289669 24298 raft_consensus.cc:374] T 0149d246c0b44093a47d45c813a47d69 P 01b6959223424811be44029b72bdac12 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:47:09.289736 24298 raft_consensus.cc:729] T 0149d246c0b44093a47d45c813a47d69 P 01b6959223424811be44029b72bdac12 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 01b6959223424811be44029b72bdac12, State: Initialized, Role: FOLLOWER
I20260925 22:47:09.290386 24298 consensus_queue.cc:260] T 0149d246c0b44093a47d45c813a47d69 P 01b6959223424811be44029b72bdac12 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01b6959223424811be44029b72bdac12" member_type: VOTER last_known_addr { host: "127.10.198.2" port: 34763 } } peers { permanent_uuid: "fea07c2152bc490580f2782c1319a37d" member_type: VOTER last_known_addr { host: "127.10.198.1" port: 36413 } } peers { permanent_uuid: "5196eab08c014dd48322cc60aa5b54bf" member_type: VOTER last_known_addr { host: "127.10.198.3" port: 37937 } }
I20260925 22:47:09.291787 24298 ts_tablet_manager.cc:1428] T 0149d246c0b44093a47d45c813a47d69 P 01b6959223424811be44029b72bdac12: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260925 22:47:09.292290 24131 heartbeater.cc:503] Master 127.10.198.62:38359 was elected leader, sending a full tablet report...
I20260925 22:47:09.292133 24302 raft_consensus.cc:348] T 0149d246c0b44093a47d45c813a47d69 P fea07c2152bc490580f2782c1319a37d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01b6959223424811be44029b72bdac12" member_type: VOTER last_known_addr { host: "127.10.198.2" port: 34763 } } peers { permanent_uuid: "fea07c2152bc490580f2782c1319a37d" member_type: VOTER last_known_addr { host: "127.10.198.1" port: 36413 } } peers { permanent_uuid: "5196eab08c014dd48322cc60aa5b54bf" member_type: VOTER last_known_addr { host: "127.10.198.3" port: 37937 } }
I20260925 22:47:09.292588 24302 raft_consensus.cc:374] T 0149d246c0b44093a47d45c813a47d69 P fea07c2152bc490580f2782c1319a37d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:47:09.292697 24302 raft_consensus.cc:729] T 0149d246c0b44093a47d45c813a47d69 P fea07c2152bc490580f2782c1319a37d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fea07c2152bc490580f2782c1319a37d, State: Initialized, Role: FOLLOWER
I20260925 22:47:09.293334 24302 consensus_queue.cc:260] T 0149d246c0b44093a47d45c813a47d69 P fea07c2152bc490580f2782c1319a37d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01b6959223424811be44029b72bdac12" member_type: VOTER last_known_addr { host: "127.10.198.2" port: 34763 } } peers { permanent_uuid: "fea07c2152bc490580f2782c1319a37d" member_type: VOTER last_known_addr { host: "127.10.198.1" port: 36413 } } peers { permanent_uuid: "5196eab08c014dd48322cc60aa5b54bf" member_type: VOTER last_known_addr { host: "127.10.198.3" port: 37937 } }
I20260925 22:47:09.296309 24302 ts_tablet_manager.cc:1428] T 0149d246c0b44093a47d45c813a47d69 P fea07c2152bc490580f2782c1319a37d: Time spent starting tablet: real 0.014s user 0.003s sys 0.004s
I20260925 22:47:09.296818 23997 heartbeater.cc:503] Master 127.10.198.62:38359 was elected leader, sending a full tablet report...
W20260925 22:47:09.336277 24132 tablet.cc:2369] T 0149d246c0b44093a47d45c813a47d69 P 01b6959223424811be44029b72bdac12: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 25 22:47:09 dist-test-slave-5vdr krb5kdc[22803](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376428, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.198.62@KRBTEST.COM for HTTP/127.10.198.20@KRBTEST.COM
I20260925 22:47:09.377655 24305 raft_consensus.cc:482] T 0149d246c0b44093a47d45c813a47d69 P 01b6959223424811be44029b72bdac12 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 22:47:09.377911 24305 raft_consensus.cc:504] T 0149d246c0b44093a47d45c813a47d69 P 01b6959223424811be44029b72bdac12 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01b6959223424811be44029b72bdac12" member_type: VOTER last_known_addr { host: "127.10.198.2" port: 34763 } } peers { permanent_uuid: "fea07c2152bc490580f2782c1319a37d" member_type: VOTER last_known_addr { host: "127.10.198.1" port: 36413 } } peers { permanent_uuid: "5196eab08c014dd48322cc60aa5b54bf" member_type: VOTER last_known_addr { host: "127.10.198.3" port: 37937 } }
I20260925 22:47:09.379873 24305 leader_election.cc:288] T 0149d246c0b44093a47d45c813a47d69 P 01b6959223424811be44029b72bdac12 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fea07c2152bc490580f2782c1319a37d (127.10.198.1:36413), 5196eab08c014dd48322cc60aa5b54bf (127.10.198.3:37937)
I20260925 22:47:09.390280 24297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:47:09.380644 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.198.2:54449 (local address 127.10.198.3:37937)
0925 22:47:09.380816 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0925 22:47:09.380823 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 22:47:09.384796 (+ 3973us) server_negotiation.cc:407] Connection header received
0925 22:47:09.385057 (+ 261us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:47:09.385061 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:47:09.385115 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:47:09.385168 (+ 53us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:47:09.385670 (+ 502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:47:09.386474 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:47:09.388800 (+ 2326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:47:09.389439 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:47:09.389564 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:47:09.389631 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0925 22:47:09.390032 (+ 401us) server_negotiation.cc:299] Negotiation successful
0925 22:47:09.390133 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260925 22:47:09.391141 24308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:47:09.380282 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.198.3:37937 (local address 127.10.198.2:54449)
0925 22:47:09.384640 (+ 4358us) negotiation.cc:107] Waiting for socket to connect
0925 22:47:09.384663 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0925 22:47:09.384905 (+ 242us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:47:09.385331 (+ 426us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:47:09.385334 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:47:09.385355 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:47:09.385521 (+ 166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:47:09.385531 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:47:09.386666 (+ 1135us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:47:09.386669 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:47:09.388596 (+ 1927us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:47:09.388616 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:47:09.389758 (+ 1142us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:47:09.389761 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:47:09.389868 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:47:09.389893 (+ 25us) client_negotiation.cc:769] Sending connection context
0925 22:47:09.390860 (+ 967us) client_negotiation.cc:240] Negotiation successful
0925 22:47:09.390964 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4253,"thread_start_us":98,"threads_started":1}
I20260925 22:47:09.392103 24219 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0149d246c0b44093a47d45c813a47d69" candidate_uuid: "01b6959223424811be44029b72bdac12" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5196eab08c014dd48322cc60aa5b54bf" is_pre_election: true
I20260925 22:47:09.392478 24219 raft_consensus.cc:2454] T 0149d246c0b44093a47d45c813a47d69 P 5196eab08c014dd48322cc60aa5b54bf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 01b6959223424811be44029b72bdac12 in term 0.
I20260925 22:47:09.393064 24021 leader_election.cc:302] T 0149d246c0b44093a47d45c813a47d69 P 01b6959223424811be44029b72bdac12 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 01b6959223424811be44029b72bdac12, 5196eab08c014dd48322cc60aa5b54bf; no voters:
I20260925 22:47:09.393394 24305 raft_consensus.cc:2781] T 0149d246c0b44093a47d45c813a47d69 P 01b6959223424811be44029b72bdac12 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 22:47:09.393483 24305 raft_consensus.cc:482] T 0149d246c0b44093a47d45c813a47d69 P 01b6959223424811be44029b72bdac12 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 22:47:09.393524 24305 raft_consensus.cc:3037] T 0149d246c0b44093a47d45c813a47d69 P 01b6959223424811be44029b72bdac12 [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:47:09.394670 24305 raft_consensus.cc:504] T 0149d246c0b44093a47d45c813a47d69 P 01b6959223424811be44029b72bdac12 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01b6959223424811be44029b72bdac12" member_type: VOTER last_known_addr { host: "127.10.198.2" port: 34763 } } peers { permanent_uuid: "fea07c2152bc490580f2782c1319a37d" member_type: VOTER last_known_addr { host: "127.10.198.1" port: 36413 } } peers { permanent_uuid: "5196eab08c014dd48322cc60aa5b54bf" member_type: VOTER last_known_addr { host: "127.10.198.3" port: 37937 } }
I20260925 22:47:09.395151 24305 leader_election.cc:288] T 0149d246c0b44093a47d45c813a47d69 P 01b6959223424811be44029b72bdac12 [CANDIDATE]: Term 1 election: Requested vote from peers fea07c2152bc490580f2782c1319a37d (127.10.198.1:36413), 5196eab08c014dd48322cc60aa5b54bf (127.10.198.3:37937)
I20260925 22:47:09.395731 24219 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0149d246c0b44093a47d45c813a47d69" candidate_uuid: "01b6959223424811be44029b72bdac12" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5196eab08c014dd48322cc60aa5b54bf"
I20260925 22:47:09.395938 24219 raft_consensus.cc:3037] T 0149d246c0b44093a47d45c813a47d69 P 5196eab08c014dd48322cc60aa5b54bf [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:47:09.397373 24219 raft_consensus.cc:2454] T 0149d246c0b44093a47d45c813a47d69 P 5196eab08c014dd48322cc60aa5b54bf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 01b6959223424811be44029b72bdac12 in term 1.
I20260925 22:47:09.398140 24021 leader_election.cc:302] T 0149d246c0b44093a47d45c813a47d69 P 01b6959223424811be44029b72bdac12 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 01b6959223424811be44029b72bdac12, 5196eab08c014dd48322cc60aa5b54bf; no voters:
I20260925 22:47:09.398337 24305 raft_consensus.cc:2781] T 0149d246c0b44093a47d45c813a47d69 P 01b6959223424811be44029b72bdac12 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 22:47:09.398629 24305 raft_consensus.cc:686] T 0149d246c0b44093a47d45c813a47d69 P 01b6959223424811be44029b72bdac12 [term 1 LEADER]: Becoming Leader. State: Replica: 01b6959223424811be44029b72bdac12, State: Running, Role: LEADER
I20260925 22:47:09.399025 24305 consensus_queue.cc:237] T 0149d246c0b44093a47d45c813a47d69 P 01b6959223424811be44029b72bdac12 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01b6959223424811be44029b72bdac12" member_type: VOTER last_known_addr { host: "127.10.198.2" port: 34763 } } peers { permanent_uuid: "fea07c2152bc490580f2782c1319a37d" member_type: VOTER last_known_addr { host: "127.10.198.1" port: 36413 } } peers { permanent_uuid: "5196eab08c014dd48322cc60aa5b54bf" member_type: VOTER last_known_addr { host: "127.10.198.3" port: 37937 } }
W20260925 22:47:09.401684 24266 tablet.cc:2369] T 0149d246c0b44093a47d45c813a47d69 P 5196eab08c014dd48322cc60aa5b54bf: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 22:47:09.403234 23799 catalog_manager.cc:5495] T 0149d246c0b44093a47d45c813a47d69 P 01b6959223424811be44029b72bdac12 reported cstate change: term changed from 0 to 1, leader changed from <none> to 01b6959223424811be44029b72bdac12 (127.10.198.2). New cstate: current_term: 1 leader_uuid: "01b6959223424811be44029b72bdac12" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01b6959223424811be44029b72bdac12" member_type: VOTER last_known_addr { host: "127.10.198.2" port: 34763 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fea07c2152bc490580f2782c1319a37d" member_type: VOTER last_known_addr { host: "127.10.198.1" port: 36413 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5196eab08c014dd48322cc60aa5b54bf" member_type: VOTER last_known_addr { host: "127.10.198.3" port: 37937 } health_report { overall_health: UNKNOWN } } }
I20260925 22:47:09.404716 24309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:47:09.380070 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.198.1:36413 (local address 127.10.198.2:57469)
0925 22:47:09.380855 (+ 785us) negotiation.cc:107] Waiting for socket to connect
0925 22:47:09.380871 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 22:47:09.381140 (+ 269us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:47:09.392857 (+ 11717us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:47:09.392860 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:47:09.392893 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:47:09.393031 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:47:09.393039 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:47:09.396936 (+ 3897us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:47:09.396940 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:47:09.403022 (+ 6082us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:47:09.403040 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:47:09.404192 (+ 1152us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:47:09.404195 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:47:09.404325 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:47:09.404351 (+ 26us) client_negotiation.cc:769] Sending connection context
0925 22:47:09.404439 (+ 88us) client_negotiation.cc:240] Negotiation successful
0925 22:47:09.404544 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":695,"thread_start_us":131,"threads_started":1}
I20260925 22:47:09.404760 24296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:47:09.380513 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.198.2:57469 (local address 127.10.198.1:36413)
0925 22:47:09.380674 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0925 22:47:09.380681 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 22:47:09.380973 (+ 292us) server_negotiation.cc:407] Connection header received
0925 22:47:09.381309 (+ 336us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:47:09.381312 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:47:09.381367 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:47:09.381420 (+ 53us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:47:09.395981 (+ 14561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:47:09.396755 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:47:09.403235 (+ 6480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:47:09.403893 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:47:09.404012 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:47:09.404070 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0925 22:47:09.404523 (+ 453us) server_negotiation.cc:299] Negotiation successful
0925 22:47:09.404628 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260925 22:47:09.405985 23933 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0149d246c0b44093a47d45c813a47d69" candidate_uuid: "01b6959223424811be44029b72bdac12" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fea07c2152bc490580f2782c1319a37d" is_pre_election: true
I20260925 22:47:09.406347 23933 raft_consensus.cc:2454] T 0149d246c0b44093a47d45c813a47d69 P fea07c2152bc490580f2782c1319a37d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 01b6959223424811be44029b72bdac12 in term 0.
I20260925 22:47:09.406941 23943 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0149d246c0b44093a47d45c813a47d69" candidate_uuid: "01b6959223424811be44029b72bdac12" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fea07c2152bc490580f2782c1319a37d"
I20260925 22:47:09.407090 23943 raft_consensus.cc:3037] T 0149d246c0b44093a47d45c813a47d69 P fea07c2152bc490580f2782c1319a37d [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:47:09.408557 23943 raft_consensus.cc:2454] T 0149d246c0b44093a47d45c813a47d69 P fea07c2152bc490580f2782c1319a37d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 01b6959223424811be44029b72bdac12 in term 1.
I20260925 22:47:09.427064 24293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:47:09.423116 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56934 (local address 127.10.198.2:34763)
0925 22:47:09.423246 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0925 22:47:09.423251 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:47:09.423274 (+ 23us) server_negotiation.cc:407] Connection header received
0925 22:47:09.423534 (+ 260us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:47:09.423537 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:47:09.423600 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:47:09.423731 (+ 131us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 22:47:09.424034 (+ 303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:47:09.424553 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:47:09.425485 (+ 932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:47:09.425714 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:47:09.425841 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:47:09.426183 (+ 342us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 22:47:09.426635 (+ 452us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 22:47:09.426788 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0925 22:47:09.426824 (+ 36us) server_negotiation.cc:299] Negotiation successful
0925 22:47:09.426890 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260925 22:47:09.434901 23943 raft_consensus.cc:1261] T 0149d246c0b44093a47d45c813a47d69 P fea07c2152bc490580f2782c1319a37d [term 1 FOLLOWER]: Refusing update from remote peer 01b6959223424811be44029b72bdac12: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 22:47:09.436685 24310 consensus_queue.cc:1035] T 0149d246c0b44093a47d45c813a47d69 P 01b6959223424811be44029b72bdac12 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fea07c2152bc490580f2782c1319a37d" member_type: VOTER last_known_addr { host: "127.10.198.1" port: 36413 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 22:47:09.437932 24219 raft_consensus.cc:1261] T 0149d246c0b44093a47d45c813a47d69 P 5196eab08c014dd48322cc60aa5b54bf [term 1 FOLLOWER]: Refusing update from remote peer 01b6959223424811be44029b72bdac12: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 22:47:09.439020 24310 consensus_queue.cc:1035] T 0149d246c0b44093a47d45c813a47d69 P 01b6959223424811be44029b72bdac12 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5196eab08c014dd48322cc60aa5b54bf" member_type: VOTER last_known_addr { host: "127.10.198.3" port: 37937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 22:47:09.456327 24316 mvcc.cc:205] Tried to move back new op lower bound from 7333381854956490752 to 7333381854822694912. Current Snapshot: MvccSnapshot[applied={T|T < 7333381854956490752}]
I20260925 22:47:09.464506 24314 mvcc.cc:205] Tried to move back new op lower bound from 7333381854956490752 to 7333381854822694912. Current Snapshot: MvccSnapshot[applied={T|T < 7333381854956490752}]
I20260925 22:47:09.467818 23798 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60738:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 22:47:09.468076 23798 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60738:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 22:47:09.471117 23798 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ec9d2e7ae7f1418caff74bbf04252560: Sending DeleteTablet for 3 replicas of tablet 0149d246c0b44093a47d45c813a47d69
I20260925 22:47:09.472124 24199 tablet_service.cc:1514] Processing DeleteTablet for tablet 0149d246c0b44093a47d45c813a47d69 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 22:47:09 UTC) from {username='kudu', principal='kudu/127.10.198.62@KRBTEST.COM'} at 127.0.0.1:41826
I20260925 22:47:09.472306 24065 tablet_service.cc:1514] Processing DeleteTablet for tablet 0149d246c0b44093a47d45c813a47d69 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 22:47:09 UTC) from {username='kudu', principal='kudu/127.10.198.62@KRBTEST.COM'} at 127.0.0.1:56924
I20260925 22:47:09.472366 23919 tablet_service.cc:1514] Processing DeleteTablet for tablet 0149d246c0b44093a47d45c813a47d69 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 22:47:09 UTC) from {username='kudu', principal='kudu/127.10.198.62@KRBTEST.COM'} at 127.0.0.1:54956
I20260925 22:47:09.473996 11032 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrWXm8X/build/debug/bin/kudu with pid 23837
I20260925 22:47:09.477197 24323 tablet_replica.cc:330] T 0149d246c0b44093a47d45c813a47d69 P 5196eab08c014dd48322cc60aa5b54bf: stopping tablet replica
I20260925 22:47:09.477617 24323 raft_consensus.cc:2229] T 0149d246c0b44093a47d45c813a47d69 P 5196eab08c014dd48322cc60aa5b54bf [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 22:47:09.478060 24323 raft_consensus.cc:2258] T 0149d246c0b44093a47d45c813a47d69 P 5196eab08c014dd48322cc60aa5b54bf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 22:47:09.478582 24324 tablet_replica.cc:330] T 0149d246c0b44093a47d45c813a47d69 P 01b6959223424811be44029b72bdac12: stopping tablet replica
I20260925 22:47:09.479027 24324 raft_consensus.cc:2229] T 0149d246c0b44093a47d45c813a47d69 P 01b6959223424811be44029b72bdac12 [term 1 LEADER]: Raft consensus shutting down.
I20260925 22:47:09.479498 24324 raft_consensus.cc:2258] T 0149d246c0b44093a47d45c813a47d69 P 01b6959223424811be44029b72bdac12 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 22:47:09.483249 24323 ts_tablet_manager.cc:1905] T 0149d246c0b44093a47d45c813a47d69 P 5196eab08c014dd48322cc60aa5b54bf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 22:47:09.486507 24324 ts_tablet_manager.cc:1905] T 0149d246c0b44093a47d45c813a47d69 P 01b6959223424811be44029b72bdac12: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 22:47:09.488039 24323 ts_tablet_manager.cc:1918] T 0149d246c0b44093a47d45c813a47d69 P 5196eab08c014dd48322cc60aa5b54bf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 22:47:09.488727 24323 log.cc:1196] T 0149d246c0b44093a47d45c813a47d69 P 5196eab08c014dd48322cc60aa5b54bf: Deleting WAL directory at /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376259650642-11032-0/minicluster-data/ts-2/wal/wals/0149d246c0b44093a47d45c813a47d69
I20260925 22:47:09.489427 24323 ts_tablet_manager.cc:1939] T 0149d246c0b44093a47d45c813a47d69 P 5196eab08c014dd48322cc60aa5b54bf: Deleting consensus metadata
I20260925 22:47:09.489948 24324 ts_tablet_manager.cc:1918] T 0149d246c0b44093a47d45c813a47d69 P 01b6959223424811be44029b72bdac12: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 22:47:09.490160 24324 log.cc:1196] T 0149d246c0b44093a47d45c813a47d69 P 01b6959223424811be44029b72bdac12: Deleting WAL directory at /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790376259650642-11032-0/minicluster-data/ts-1/wal/wals/0149d246c0b44093a47d45c813a47d69
I20260925 22:47:09.490648 24324 ts_tablet_manager.cc:1939] T 0149d246c0b44093a47d45c813a47d69 P 01b6959223424811be44029b72bdac12: Deleting consensus metadata
I20260925 22:47:09.491736 23786 catalog_manager.cc:4841] TS 5196eab08c014dd48322cc60aa5b54bf (127.10.198.3:37937): tablet 0149d246c0b44093a47d45c813a47d69 (table test-table [id=559b38368de9403db2c6a0dc6aa1b6fc]) successfully deleted
I20260925 22:47:09.491771 23783 catalog_manager.cc:4841] TS 01b6959223424811be44029b72bdac12 (127.10.198.2:34763): tablet 0149d246c0b44093a47d45c813a47d69 (table test-table [id=559b38368de9403db2c6a0dc6aa1b6fc]) successfully deleted
I20260925 22:47:09.492436 11032 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrWXm8X/build/debug/bin/kudu with pid 24001
W20260925 22:47:09.492447 23783 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.10.198.1:36413 (error 108)
I20260925 22:47:09.494794 24295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:47:09.494525 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.198.1:36413 (local address 127.0.0.1:54964)
0925 22:47:09.494647 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0925 22:47:09.494715 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.198.1:36413: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260925 22:47:09.495124 23783 catalog_manager.cc:4584] TS fea07c2152bc490580f2782c1319a37d (127.10.198.1:36413): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 0149d246c0b44093a47d45c813a47d69: Network error: Client connection negotiation failed: client connection to 127.10.198.1:36413: connect: Connection refused (error 111)
I20260925 22:47:09.499164 11032 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrWXm8X/build/debug/bin/kudu with pid 24134
I20260925 22:47:09.508559 11032 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrWXm8X/build/debug/bin/kudu with pid 23766
I20260925 22:47:09.521951 11032 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260925 22:47:09.868569 11032 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260925 22:47:09.868692 11032 mini_ranger.cc:66] Stopping Ranger...
I20260925 22:47:10.215037 11032 mini_ranger.cc:68] Stopped Ranger
2026-09-25 22:47:10.215 UTC [22833] LOG: received smart shutdown request
2026-09-25 22:47:10.219 UTC [22833] LOG: background worker "logical replication launcher" (PID 22842) exited with exit code 1
2026-09-25 22:47:10.220 UTC [22837] LOG: shutting down
2026-09-25 22:47:10.382 UTC [22833] LOG: database system is shut down
2026-09-25T22:47:10Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (54249 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790376259650642-11032-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 22:47:10 dist-test-slave-5vdr krb5kdc[24336](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 22:47:10 dist-test-slave-5vdr krb5kdc[24336](info): set up 2 sockets
Sep 25 22:47:10 dist-test-slave-5vdr krb5kdc[24336](info): commencing operation
krb5kdc: starting...
W20260925 22:47:11.564042 11032 mini_kdc.cc:121] Time spent starting KDC: real 1.035s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790376259650642-11032-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790376259650642-11032-0/krb5kdc/test-user.keytab.
Sep 25 22:47:11 dist-test-slave-5vdr krb5kdc[24336](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376431, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T22:47:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T22:47:11Z Disabled control of system clock
WARNING: no policy specified for kudu/127.10.198.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.198.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.198.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790376259650642-11032-0/krb5kdc/kudu_127.10.198.62.keytab.
Entry for principal kudu/127.10.198.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790376259650642-11032-0/krb5kdc/kudu_127.10.198.62.keytab.
I20260925 22:47:11.695362 11032 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrWXm8X/build/debug/bin/kudu
/tmp/dist-test-taskrWXm8X/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790376259650642-11032-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790376259650642-11032-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790376259650642-11032-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790376259650642-11032-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.198.62:36499
--webserver_interface=127.10.198.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790376259650642-11032-0/krb5kdc/kudu_127.10.198.62.keytab
--principal=kudu/127.10.198.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.198.20:37927
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.198.62:36499,127.10.198.61:40537,127.10.198.60:39445
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790376259650642-11032-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790376259650642-11032-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790376259650642-11032-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:47:11.815527 24349 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:47:11.815984 24349 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:47:11.816097 24349 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:47:11.820487 24349 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 22:47:11.820621 24349 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:47:11.820657 24349 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:47:11.820698 24349 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 22:47:11.820730 24349 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 22:47:11.826622 24349 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.198.20:37927
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790376259650642-11032-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790376259650642-11032-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.198.62:36499,127.10.198.61:40537,127.10.198.60:39445
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790376259650642-11032-0/krb5kdc/kudu_127.10.198.62.keytab
--principal=kudu/127.10.198.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.198.62:36499
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790376259650642-11032-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.198.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24349
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790376259650642-11032-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:09:16 UTC on ccycloud.quasar-afmhyw.root.comops.site
build id 4673379
I20260925 22:47:11.827943 24349 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:47:11.829069 24349 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:47:11.838421 24357 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:47:11.838421 24355 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:47:11.838421 24354 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:47:11.838601 24349 server_base.cc:1034] running on GCE node
I20260925 22:47:11.839550 24349 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:47:11.841114 24349 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:47:11.842381 24349 hybrid_clock.cc:648] HybridClock initialized: now 1790376431842373 us; error 49 us; skew 500 ppm
Sep 25 22:47:11 dist-test-slave-5vdr krb5kdc[24336](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376431, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.198.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:47:11.845759 24349 init.cc:377] Logged in from keytab as kudu/127.10.198.62@KRBTEST.COM (short username kudu)
I20260925 22:47:11.847043 24349 webserver.cc:460] Webserver started at http://127.10.198.62:36135/ using document root <none> and password file <none>
I20260925 22:47:11.847697 24349 fs_manager.cc:359] Metadata directory not provided
I20260925 22:47:11.847802 24349 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:47:11.848078 24349 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:47:11.850477 24349 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790376259650642-11032-0/minicluster-data/master-0/data/instance:
uuid: "2bffee41198446adab3908db924fa468"
format_stamp: "Formatted at 2026-09-25 22:47:11 on dist-test-slave-5vdr"
server_key: "6cc323715ee823ae3e33aa9d9c12b8de"
server_key_iv: "1f533d8d3547e243a892c9515e41b903"
server_key_version: "encryptionkey@0"
I20260925 22:47:11.851150 24349 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790376259650642-11032-0/minicluster-data/master-0/wal/instance:
uuid: "2bffee41198446adab3908db924fa468"
format_stamp: "Formatted at 2026-09-25 22:47:11 on dist-test-slave-5vdr"
server_key: "6cc323715ee823ae3e33aa9d9c12b8de"
server_key_iv: "1f533d8d3547e243a892c9515e41b903"
server_key_version: "encryptionkey@0"
I20260925 22:47:11.855594 24349 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260925 22:47:11.858603 24364 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:47:11.860076 24349 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 22:47:11.860286 24349 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790376259650642-11032-0/minicluster-data/master-0/data,/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790376259650642-11032-0/minicluster-data/master-0/wal
uuid: "2bffee41198446adab3908db924fa468"
format_stamp: "Formatted at 2026-09-25 22:47:11 on dist-test-slave-5vdr"
server_key: "6cc323715ee823ae3e33aa9d9c12b8de"
server_key_iv: "1f533d8d3547e243a892c9515e41b903"
server_key_version: "encryptionkey@0"
I20260925 22:47:11.860458 24349 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790376259650642-11032-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790376259650642-11032-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790376259650642-11032-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:47:11.871939 24349 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:47:11.875793 24349 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:47:11.876068 24349 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:47:11.884178 24349 rpc_server.cc:307] RPC server started. Bound to: 127.10.198.62:36499
I20260925 22:47:11.884198 24416 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.198.62:36499 every 8 connection(s)
I20260925 22:47:11.885263 24349 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790376259650642-11032-0/minicluster-data/master-0/data/info.pb
I20260925 22:47:11.888468 24417 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:47:11.892954 11032 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrWXm8X/build/debug/bin/kudu as pid 24349
I20260925 22:47:11.893092 11032 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790376259650642-11032-0/minicluster-data/master-0/wal/instance
I20260925 22:47:11.892746 24417 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.198.62" port: 36499 } has no permanent_uuid. Determining permanent_uuid...
I20260925 22:47:11.893383 11032 external_mini_cluster.cc:1428] Setting key 46e9095b74c20984141980b7b63892f4
WARNING: no policy specified for kudu/127.10.198.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.198.61@KRBTEST.COM" created.
Sep 25 22:47:11 dist-test-slave-5vdr krb5kdc[24336](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376431, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.198.62@KRBTEST.COM for kudu/127.10.198.62@KRBTEST.COM
I20260925 22:47:11.911072 24420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:47:11.896068 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37900 (local address 127.10.198.62:36499)
0925 22:47:11.896582 (+ 514us) server_negotiation.cc:206] Beginning negotiation
0925 22:47:11.896603 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0925 22:47:11.896912 (+ 309us) server_negotiation.cc:407] Connection header received
0925 22:47:11.898090 (+ 1178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:47:11.898095 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:47:11.898295 (+ 200us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:47:11.898505 (+ 210us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:47:11.899435 (+ 930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:47:11.900451 (+ 1016us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:47:11.901555 (+ 1104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:47:11.902039 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:47:11.902253 (+ 214us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:47:11.905423 (+ 3170us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:47:11.905434 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:47:11.905454 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:47:11.905494 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:47:11.907447 (+ 1953us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:47:11.908200 (+ 753us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:47:11.908206 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:47:11.908207 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:47:11.908328 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:47:11.908783 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:47:11.908786 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:47:11.908787 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:47:11.909227 (+ 440us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:47:11.909339 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0925 22:47:11.909901 (+ 562us) server_negotiation.cc:299] Negotiation successful
0925 22:47:11.910154 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":4,"server-negotiator.queue_time_us":340,"thread_start_us":153,"threads_started":1}
I20260925 22:47:11.911078 24421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:47:11.896068 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.198.62:36499 (local address 127.0.0.1:37900)
0925 22:47:11.896740 (+ 672us) negotiation.cc:107] Waiting for socket to connect
0925 22:47:11.896782 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0925 22:47:11.897736 (+ 954us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:47:11.898504 (+ 768us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:47:11.898510 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:47:11.898992 (+ 482us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:47:11.899254 (+ 262us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:47:11.899269 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:47:11.900591 (+ 1322us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:47:11.900595 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:47:11.901415 (+ 820us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:47:11.901425 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:47:11.902231 (+ 806us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:47:11.902235 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:47:11.902349 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:47:11.903038 (+ 689us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:47:11.903057 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:47:11.905202 (+ 2145us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:47:11.907620 (+ 2418us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:47:11.907635 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:47:11.907650 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:47:11.908076 (+ 426us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:47:11.908445 (+ 369us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:47:11.908451 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:47:11.908456 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:47:11.908651 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:47:11.909364 (+ 713us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:47:11.909374 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:47:11.909592 (+ 218us) client_negotiation.cc:769] Sending connection context
0925 22:47:11.909816 (+ 224us) client_negotiation.cc:240] Negotiation successful
0925 22:47:11.910264 (+ 448us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":478,"mutex_wait_us":62,"spinlock_wait_cycles":768,"thread_start_us":188,"threads_started":1}
I20260925 22:47:11.914213 24417 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.198.61" port: 40537 } has no permanent_uuid. Determining permanent_uuid...
I20260925 22:47:11.915170 24421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:47:11.914844 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.198.61:40537 (local address 127.0.0.1:52450)
0925 22:47:11.914996 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0925 22:47:11.915089 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.198.61:40537: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
W20260925 22:47:11.915436 24366 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.10.198.61:40537: connect: Connection refused (error 111)
I20260925 22:47:11.916540 24421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:47:11.916285 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.198.61:40537 (local address 127.0.0.1:52458)
0925 22:47:11.916414 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0925 22:47:11.916470 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.198.61:40537: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260925 22:47:11.916740 24417 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.10.198.61:40537: Network error: Client connection negotiation failed: client connection to 127.10.198.61:40537: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.198.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790376259650642-11032-0/krb5kdc/kudu_127.10.198.61.keytab.
Entry for principal kudu/127.10.198.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790376259650642-11032-0/krb5kdc/kudu_127.10.198.61.keytab.
I20260925 22:47:11.923913 11032 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrWXm8X/build/debug/bin/kudu
/tmp/dist-test-taskrWXm8X/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790376259650642-11032-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790376259650642-11032-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790376259650642-11032-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790376259650642-11032-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.198.61:40537
--webserver_interface=127.10.198.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790376259650642-11032-0/krb5kdc/kudu_127.10.198.61.keytab
--principal=kudu/127.10.198.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.198.20:37927
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.198.62:36499,127.10.198.61:40537,127.10.198.60:39445
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790376259650642-11032-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790376259650642-11032-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790376259650642-11032-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 22:47:11.966095 24417 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.198.61" port: 40537 } attempt: 1
I20260925 22:47:11.967048 24421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:47:11.966738 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.198.61:40537 (local address 127.0.0.1:52466)
0925 22:47:11.966899 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0925 22:47:11.966966 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.198.61:40537: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260925 22:47:11.967958 24421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:47:11.967634 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.198.61:40537 (local address 127.0.0.1:52472)
0925 22:47:11.967767 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0925 22:47:11.967827 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.198.61:40537: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260925 22:47:11.968099 24417 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.10.198.61:40537: Network error: Client connection negotiation failed: client connection to 127.10.198.61:40537: connect: Connection refused (error 111)
I20260925 22:47:12.036290 24417 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.198.61" port: 40537 } attempt: 2
I20260925 22:47:12.037261 24421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:47:12.036897 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.198.61:40537 (local address 127.0.0.1:52478)
0925 22:47:12.037098 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0925 22:47:12.037163 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.198.61:40537: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87}
I20260925 22:47:12.038039 24421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:47:12.037794 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.198.61:40537 (local address 127.0.0.1:52482)
0925 22:47:12.037925 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0925 22:47:12.037978 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.198.61:40537: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260925 22:47:12.038158 24417 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.10.198.61:40537: Network error: Client connection negotiation failed: client connection to 127.10.198.61:40537: connect: Connection refused (error 111)
W20260925 22:47:12.043440 24424 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:47:12.043839 24424 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:47:12.043946 24424 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:47:12.048578 24424 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 22:47:12.048691 24424 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:47:12.048727 24424 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:47:12.048786 24424 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 22:47:12.048825 24424 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 22:47:12.054791 24424 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.198.20:37927
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790376259650642-11032-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790376259650642-11032-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.10.198.62:36499,127.10.198.61:40537,127.10.198.60:39445
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790376259650642-11032-0/krb5kdc/kudu_127.10.198.61.keytab
--principal=kudu/127.10.198.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.198.61:40537
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790376259650642-11032-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.10.198.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24424
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790376259650642-11032-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:09:16 UTC on ccycloud.quasar-afmhyw.root.comops.site
build id 4673379
I20260925 22:47:12.056205 24424 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:47:12.057379 24424 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:47:12.066951 24432 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:47:12.066874 24430 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:47:12.067200 24424 server_base.cc:1034] running on GCE node
W20260925 22:47:12.066874 24429 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:47:12.068019 24424 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:47:12.069278 24424 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:47:12.070503 24424 hybrid_clock.cc:648] HybridClock initialized: now 1790376432070462 us; error 60 us; skew 500 ppm
Sep 25 22:47:12 dist-test-slave-5vdr krb5kdc[24336](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376432, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.198.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:47:12.073822 24424 init.cc:377] Logged in from keytab as kudu/127.10.198.61@KRBTEST.COM (short username kudu)
I20260925 22:47:12.074929 24424 webserver.cc:460] Webserver started at http://127.10.198.61:46279/ using document root <none> and password file <none>
I20260925 22:47:12.075577 24424 fs_manager.cc:359] Metadata directory not provided
I20260925 22:47:12.075687 24424 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:47:12.076090 24424 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:47:12.078665 24424 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790376259650642-11032-0/minicluster-data/master-1/data/instance:
uuid: "cabf5b6801764facb094d08b9cd902db"
format_stamp: "Formatted at 2026-09-25 22:47:12 on dist-test-slave-5vdr"
server_key: "99cbced0db04502945bda43800acbf4c"
server_key_iv: "4c246f463dc79e6e304f994850ef3dad"
server_key_version: "encryptionkey@0"
I20260925 22:47:12.079319 24424 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790376259650642-11032-0/minicluster-data/master-1/wal/instance:
uuid: "cabf5b6801764facb094d08b9cd902db"
format_stamp: "Formatted at 2026-09-25 22:47:12 on dist-test-slave-5vdr"
server_key: "99cbced0db04502945bda43800acbf4c"
server_key_iv: "4c246f463dc79e6e304f994850ef3dad"
server_key_version: "encryptionkey@0"
I20260925 22:47:12.083948 24424 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.004s
I20260925 22:47:12.087387 24439 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:47:12.089226 24424 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260925 22:47:12.089450 24424 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790376259650642-11032-0/minicluster-data/master-1/data,/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790376259650642-11032-0/minicluster-data/master-1/wal
uuid: "cabf5b6801764facb094d08b9cd902db"
format_stamp: "Formatted at 2026-09-25 22:47:12 on dist-test-slave-5vdr"
server_key: "99cbced0db04502945bda43800acbf4c"
server_key_iv: "4c246f463dc79e6e304f994850ef3dad"
server_key_version: "encryptionkey@0"
I20260925 22:47:12.089607 24424 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790376259650642-11032-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790376259650642-11032-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790376259650642-11032-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:47:12.110737 24424 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:47:12.114677 24424 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:47:12.114925 24424 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:47:12.123813 24424 rpc_server.cc:307] RPC server started. Bound to: 127.10.198.61:40537
I20260925 22:47:12.123986 24491 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.198.61:40537 every 8 connection(s)
I20260925 22:47:12.125255 24424 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790376259650642-11032-0/minicluster-data/master-1/data/info.pb
I20260925 22:47:12.128770 24492 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:47:12.129333 24417 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.198.61" port: 40537 } attempt: 3
I20260925 22:47:12.131517 11032 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrWXm8X/build/debug/bin/kudu as pid 24424
I20260925 22:47:12.131666 11032 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790376259650642-11032-0/minicluster-data/master-1/wal/instance
I20260925 22:47:12.132037 11032 external_mini_cluster.cc:1428] Setting key b3e1e4faf12e7a036f978e122a869566
I20260925 22:47:12.136113 24492 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.198.62" port: 36499 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 22:47:12 dist-test-slave-5vdr krb5kdc[24336](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376431, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.198.62@KRBTEST.COM for kudu/127.10.198.61@KRBTEST.COM
WARNING: no policy specified for kudu/127.10.198.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.198.60@KRBTEST.COM" created.
I20260925 22:47:12.146092 24421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:47:12.130081 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.198.61:40537 (local address 127.0.0.1:52484)
0925 22:47:12.130357 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0925 22:47:12.130376 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0925 22:47:12.130518 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:47:12.134351 (+ 3833us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:47:12.134355 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:47:12.134606 (+ 251us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:47:12.134756 (+ 150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:47:12.134765 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:47:12.136272 (+ 1507us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:47:12.136279 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:47:12.136917 (+ 638us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:47:12.136931 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:47:12.137846 (+ 915us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:47:12.137850 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:47:12.137972 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:47:12.138592 (+ 620us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:47:12.138616 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:47:12.140496 (+ 1880us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:47:12.143329 (+ 2833us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:47:12.143334 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:47:12.143336 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:47:12.143594 (+ 258us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:47:12.144116 (+ 522us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:47:12.144119 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:47:12.144121 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:47:12.144219 (+ 98us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:47:12.145442 (+ 1223us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:47:12.145449 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:47:12.145591 (+ 142us) client_negotiation.cc:769] Sending connection context
0925 22:47:12.145738 (+ 147us) client_negotiation.cc:240] Negotiation successful
0925 22:47:12.145912 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":169}
Sep 25 22:47:12 dist-test-slave-5vdr krb5kdc[24336](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376432, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.198.61@KRBTEST.COM for kudu/127.10.198.62@KRBTEST.COM
I20260925 22:47:12.147748 24494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:47:12.131178 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52484 (local address 127.10.198.61:40537)
0925 22:47:12.131779 (+ 601us) server_negotiation.cc:206] Beginning negotiation
0925 22:47:12.131803 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0925 22:47:12.131928 (+ 125us) server_negotiation.cc:407] Connection header received
0925 22:47:12.133145 (+ 1217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:47:12.133180 (+ 35us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:47:12.133787 (+ 607us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:47:12.134406 (+ 619us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:47:12.134959 (+ 553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:47:12.136107 (+ 1148us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:47:12.137074 (+ 967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:47:12.137688 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:47:12.137881 (+ 193us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:47:12.140716 (+ 2835us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:47:12.140723 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:47:12.140740 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:47:12.140781 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:47:12.143139 (+ 2358us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:47:12.143738 (+ 599us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:47:12.143749 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:47:12.143757 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:47:12.143976 (+ 219us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:47:12.144366 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:47:12.144370 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:47:12.144371 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:47:12.145206 (+ 835us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:47:12.145352 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0925 22:47:12.145936 (+ 584us) server_negotiation.cc:299] Negotiation successful
0925 22:47:12.146498 (+ 562us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":397,"thread_start_us":185,"threads_started":1}
I20260925 22:47:12.150554 24417 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.198.60" port: 39445 } has no permanent_uuid. Determining permanent_uuid...
I20260925 22:47:12.151437 24421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:47:12.151150 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.198.60:39445 (local address 127.0.0.1:37596)
0925 22:47:12.151289 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0925 22:47:12.151348 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.198.60:39445: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260925 22:47:12.152329 24421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:47:12.152082 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.198.60:39445 (local address 127.0.0.1:37610)
0925 22:47:12.152211 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0925 22:47:12.152258 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.198.60:39445: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
W20260925 22:47:12.152582 24417 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.10.198.60:39445: Network error: Client connection negotiation failed: client connection to 127.10.198.60:39445: connect: Connection refused (error 111)
I20260925 22:47:12.152864 24420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:47:12.138849 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37910 (local address 127.10.198.62:36499)
0925 22:47:12.138997 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0925 22:47:12.139003 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 22:47:12.139444 (+ 441us) server_negotiation.cc:407] Connection header received
0925 22:47:12.139657 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:47:12.139660 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:47:12.139720 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:47:12.139778 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:47:12.140842 (+ 1064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:47:12.141745 (+ 903us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:47:12.142831 (+ 1086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:47:12.143988 (+ 1157us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:47:12.144102 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:47:12.147567 (+ 3465us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:47:12.147572 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:47:12.147574 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:47:12.147602 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:47:12.150408 (+ 2806us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:47:12.151092 (+ 684us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:47:12.151098 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:47:12.151104 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:47:12.151221 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:47:12.151673 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:47:12.151677 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:47:12.151679 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:47:12.151993 (+ 314us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:47:12.152089 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0925 22:47:12.152621 (+ 532us) server_negotiation.cc:299] Negotiation successful
0925 22:47:12.152770 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64,"spinlock_wait_cycles":6528}
I20260925 22:47:12.153030 24496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:47:12.138843 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.198.62:36499 (local address 127.0.0.1:37910)
0925 22:47:12.139308 (+ 465us) negotiation.cc:107] Waiting for socket to connect
0925 22:47:12.139366 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0925 22:47:12.139543 (+ 177us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:47:12.140032 (+ 489us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:47:12.140036 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:47:12.140484 (+ 448us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:47:12.140699 (+ 215us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:47:12.140707 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:47:12.141919 (+ 1212us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:47:12.141926 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:47:12.142685 (+ 759us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:47:12.142696 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:47:12.144262 (+ 1566us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:47:12.144290 (+ 28us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:47:12.144537 (+ 247us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:47:12.145272 (+ 735us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:47:12.145293 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:47:12.147333 (+ 2040us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:47:12.150543 (+ 3210us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:47:12.150554 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:47:12.150560 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:47:12.150958 (+ 398us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:47:12.151367 (+ 409us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:47:12.151373 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:47:12.151378 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:47:12.151508 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:47:12.152140 (+ 632us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:47:12.152151 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:47:12.152327 (+ 176us) client_negotiation.cc:769] Sending connection context
0925 22:47:12.152522 (+ 195us) client_negotiation.cc:240] Negotiation successful
0925 22:47:12.152794 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":288,"thread_start_us":180,"threads_started":1}
I20260925 22:47:12.154621 24492 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.198.61" port: 40537 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 22:47:12 dist-test-slave-5vdr krb5kdc[24336](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376432, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.198.61@KRBTEST.COM for kudu/127.10.198.61@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.198.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790376259650642-11032-0/krb5kdc/kudu_127.10.198.60.keytab.
Entry for principal kudu/127.10.198.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790376259650642-11032-0/krb5kdc/kudu_127.10.198.60.keytab.
I20260925 22:47:12.164171 11032 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrWXm8X/build/debug/bin/kudu
/tmp/dist-test-taskrWXm8X/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790376259650642-11032-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790376259650642-11032-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790376259650642-11032-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790376259650642-11032-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.198.60:39445
--webserver_interface=127.10.198.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790376259650642-11032-0/krb5kdc/kudu_127.10.198.60.keytab
--principal=kudu/127.10.198.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.198.20:37927
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.198.62:36499,127.10.198.61:40537,127.10.198.60:39445
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790376259650642-11032-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790376259650642-11032-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790376259650642-11032-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 22:47:12.165768 24496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:47:12.155291 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.198.61:40537 (local address 127.0.0.1:52492)
0925 22:47:12.155447 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0925 22:47:12.155463 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 22:47:12.155562 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:47:12.155880 (+ 318us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:47:12.155883 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:47:12.156104 (+ 221us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:47:12.156222 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:47:12.156228 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:47:12.157151 (+ 923us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:47:12.157154 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:47:12.158054 (+ 900us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:47:12.158067 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:47:12.158635 (+ 568us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:47:12.158638 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:47:12.158729 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:47:12.159439 (+ 710us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:47:12.159452 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:47:12.160988 (+ 1536us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:47:12.163609 (+ 2621us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:47:12.163612 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:47:12.163614 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:47:12.164089 (+ 475us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:47:12.164473 (+ 384us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:47:12.164479 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:47:12.164484 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:47:12.164579 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:47:12.165041 (+ 462us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:47:12.165047 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:47:12.165176 (+ 129us) client_negotiation.cc:769] Sending connection context
0925 22:47:12.165293 (+ 117us) client_negotiation.cc:240] Negotiation successful
0925 22:47:12.165502 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":61}
I20260925 22:47:12.165892 24494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:47:12.155312 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52492 (local address 127.10.198.61:40537)
0925 22:47:12.155469 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0925 22:47:12.155473 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:47:12.155515 (+ 42us) server_negotiation.cc:407] Connection header received
0925 22:47:12.155673 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:47:12.155676 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:47:12.155720 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:47:12.156014 (+ 294us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:47:12.156336 (+ 322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:47:12.156974 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:47:12.158205 (+ 1231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:47:12.158461 (+ 256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:47:12.158622 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:47:12.161184 (+ 2562us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:47:12.161187 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:47:12.161189 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:47:12.161212 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:47:12.163385 (+ 2173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:47:12.164213 (+ 828us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:47:12.164217 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:47:12.164219 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:47:12.164324 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:47:12.164672 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:47:12.164675 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:47:12.164676 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:47:12.164933 (+ 257us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:47:12.165020 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0925 22:47:12.165455 (+ 435us) server_negotiation.cc:299] Negotiation successful
0925 22:47:12.165627 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260925 22:47:12.166814 24492 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.198.60" port: 39445 } has no permanent_uuid. Determining permanent_uuid...
I20260925 22:47:12.167721 24496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:47:12.167396 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.198.60:39445 (local address 127.0.0.1:37612)
0925 22:47:12.167545 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0925 22:47:12.167648 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.198.60:39445: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
W20260925 22:47:12.168548 24440 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.10.198.60:39445: connect: Connection refused (error 111)
I20260925 22:47:12.169836 24496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:47:12.169482 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.198.60:39445 (local address 127.0.0.1:37620)
0925 22:47:12.169678 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0925 22:47:12.169751 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.198.60:39445: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
W20260925 22:47:12.170046 24492 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.10.198.60:39445: Network error: Client connection negotiation failed: client connection to 127.10.198.60:39445: connect: Connection refused (error 111)
I20260925 22:47:12.183797 24417 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.198.60" port: 39445 } attempt: 1
I20260925 22:47:12.184840 24421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:47:12.184465 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.198.60:39445 (local address 127.0.0.1:37622)
0925 22:47:12.184660 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0925 22:47:12.184736 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.198.60:39445: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
I20260925 22:47:12.185727 24421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:47:12.185434 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.198.60:39445 (local address 127.0.0.1:37636)
0925 22:47:12.185582 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0925 22:47:12.185647 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.198.60:39445: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
W20260925 22:47:12.185902 24417 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.10.198.60:39445: Network error: Client connection negotiation failed: client connection to 127.10.198.60:39445: connect: Connection refused (error 111)
I20260925 22:47:12.219369 24492 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.198.60" port: 39445 } attempt: 1
I20260925 22:47:12.220513 24496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:47:12.220145 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.198.60:39445 (local address 127.0.0.1:37640)
0925 22:47:12.220336 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0925 22:47:12.220415 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.198.60:39445: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
I20260925 22:47:12.221423 24496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:47:12.221174 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.198.60:39445 (local address 127.0.0.1:37652)
0925 22:47:12.221294 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0925 22:47:12.221351 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.198.60:39445: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260925 22:47:12.221601 24492 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.10.198.60:39445: Network error: Client connection negotiation failed: client connection to 127.10.198.60:39445: connect: Connection refused (error 111)
I20260925 22:47:12.261230 24417 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.198.60" port: 39445 } attempt: 2
I20260925 22:47:12.262218 24421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:47:12.261869 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.198.60:39445 (local address 127.0.0.1:37666)
0925 22:47:12.262065 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0925 22:47:12.262133 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.198.60:39445: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":86}
I20260925 22:47:12.263159 24421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:47:12.262882 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.198.60:39445 (local address 127.0.0.1:37668)
0925 22:47:12.263038 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0925 22:47:12.263091 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.198.60:39445: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
W20260925 22:47:12.263309 24417 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.10.198.60:39445: Network error: Client connection negotiation failed: client connection to 127.10.198.60:39445: connect: Connection refused (error 111)
I20260925 22:47:12.289813 24492 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.198.60" port: 39445 } attempt: 2
I20260925 22:47:12.290751 24496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:47:12.290440 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.198.60:39445 (local address 127.0.0.1:37680)
0925 22:47:12.290603 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0925 22:47:12.290669 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.198.60:39445: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
I20260925 22:47:12.291592 24496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:47:12.291347 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.198.60:39445 (local address 127.0.0.1:37694)
0925 22:47:12.291467 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0925 22:47:12.291527 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.198.60:39445: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260925 22:47:12.291730 24492 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.10.198.60:39445: Network error: Client connection negotiation failed: client connection to 127.10.198.60:39445: connect: Connection refused (error 111)
W20260925 22:47:12.293226 24498 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:47:12.293565 24498 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:47:12.293669 24498 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:47:12.298168 24498 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 22:47:12.298285 24498 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:47:12.298326 24498 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:47:12.298389 24498 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 22:47:12.298475 24498 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 22:47:12.304606 24498 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.198.20:37927
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790376259650642-11032-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790376259650642-11032-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.10.198.62:36499,127.10.198.61:40537,127.10.198.60:39445
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790376259650642-11032-0/krb5kdc/kudu_127.10.198.60.keytab
--principal=kudu/127.10.198.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.198.60:39445
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790376259650642-11032-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.10.198.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24498
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790376259650642-11032-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:09:16 UTC on ccycloud.quasar-afmhyw.root.comops.site
build id 4673379
I20260925 22:47:12.305833 24498 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:47:12.306962 24498 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:47:12.316231 24504 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:47:12.316231 24505 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:47:12.316255 24507 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:47:12.316551 24498 server_base.cc:1034] running on GCE node
I20260925 22:47:12.317188 24498 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:47:12.318513 24498 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:47:12.319734 24498 hybrid_clock.cc:648] HybridClock initialized: now 1790376432319714 us; error 43 us; skew 500 ppm
Sep 25 22:47:12 dist-test-slave-5vdr krb5kdc[24336](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376432, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.198.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:47:12.323022 24498 init.cc:377] Logged in from keytab as kudu/127.10.198.60@KRBTEST.COM (short username kudu)
I20260925 22:47:12.324219 24498 webserver.cc:460] Webserver started at http://127.10.198.60:44315/ using document root <none> and password file <none>
I20260925 22:47:12.324891 24498 fs_manager.cc:359] Metadata directory not provided
I20260925 22:47:12.325001 24498 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:47:12.325255 24498 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:47:12.327535 24498 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790376259650642-11032-0/minicluster-data/master-2/data/instance:
uuid: "c296129fe03540e89de350f61469a203"
format_stamp: "Formatted at 2026-09-25 22:47:12 on dist-test-slave-5vdr"
server_key: "34a259c1ec8bc9af2363e960fccefc01"
server_key_iv: "38fc1ace1fc116aae3ba268628515a53"
server_key_version: "encryptionkey@0"
I20260925 22:47:12.328269 24498 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790376259650642-11032-0/minicluster-data/master-2/wal/instance:
uuid: "c296129fe03540e89de350f61469a203"
format_stamp: "Formatted at 2026-09-25 22:47:12 on dist-test-slave-5vdr"
server_key: "34a259c1ec8bc9af2363e960fccefc01"
server_key_iv: "38fc1ace1fc116aae3ba268628515a53"
server_key_version: "encryptionkey@0"
I20260925 22:47:12.332705 24498 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260925 22:47:12.335592 24514 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:47:12.337143 24498 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260925 22:47:12.337356 24498 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790376259650642-11032-0/minicluster-data/master-2/data,/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790376259650642-11032-0/minicluster-data/master-2/wal
uuid: "c296129fe03540e89de350f61469a203"
format_stamp: "Formatted at 2026-09-25 22:47:12 on dist-test-slave-5vdr"
server_key: "34a259c1ec8bc9af2363e960fccefc01"
server_key_iv: "38fc1ace1fc116aae3ba268628515a53"
server_key_version: "encryptionkey@0"
I20260925 22:47:12.337525 24498 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790376259650642-11032-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790376259650642-11032-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790376259650642-11032-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:47:12.358383 24498 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:47:12.361683 24498 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:47:12.361927 24498 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:47:12.362478 24417 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.198.60" port: 39445 } attempt: 3
I20260925 22:47:12.363360 24421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:47:12.363014 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.198.60:39445 (local address 127.0.0.1:37704)
0925 22:47:12.363160 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0925 22:47:12.363228 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.198.60:39445: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260925 22:47:12.364193 24421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:47:12.363960 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.198.60:39445 (local address 127.0.0.1:37712)
0925 22:47:12.364077 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0925 22:47:12.364132 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.198.60:39445: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260925 22:47:12.364327 24417 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.10.198.60:39445: Network error: Client connection negotiation failed: client connection to 127.10.198.60:39445: connect: Connection refused (error 111)
I20260925 22:47:12.370395 24498 rpc_server.cc:307] RPC server started. Bound to: 127.10.198.60:39445
I20260925 22:47:12.370422 24566 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.198.60:39445 every 8 connection(s)
I20260925 22:47:12.371501 24498 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790376259650642-11032-0/minicluster-data/master-2/data/info.pb
I20260925 22:47:12.372807 11032 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrWXm8X/build/debug/bin/kudu as pid 24498
I20260925 22:47:12.372961 11032 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790376259650642-11032-0/minicluster-data/master-2/wal/instance
I20260925 22:47:12.373237 11032 external_mini_cluster.cc:1428] Setting key 1e8873ebc6a1e3850949c34ad6e4d62b
I20260925 22:47:12.375090 24567 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:47:12.380081 24567 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.198.62" port: 36499 } has no permanent_uuid. Determining permanent_uuid...
I20260925 22:47:12.382903 24492 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.198.60" port: 39445 } attempt: 3
Sep 25 22:47:12 dist-test-slave-5vdr krb5kdc[24336](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376431, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.198.60@KRBTEST.COM
Sep 25 22:47:12 dist-test-slave-5vdr krb5kdc[24336](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376432, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.198.61@KRBTEST.COM for kudu/127.10.198.60@KRBTEST.COM
Sep 25 22:47:12 dist-test-slave-5vdr krb5kdc[24336](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376432, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.198.60@KRBTEST.COM for kudu/127.10.198.62@KRBTEST.COM
I20260925 22:47:12.391664 24570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:47:12.374686 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37720 (local address 127.10.198.60:39445)
0925 22:47:12.375157 (+ 471us) server_negotiation.cc:206] Beginning negotiation
0925 22:47:12.375174 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0925 22:47:12.375237 (+ 63us) server_negotiation.cc:407] Connection header received
0925 22:47:12.376024 (+ 787us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:47:12.376046 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:47:12.376417 (+ 371us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:47:12.376731 (+ 314us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:47:12.377833 (+ 1102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:47:12.379078 (+ 1245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:47:12.380379 (+ 1301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:47:12.381030 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:47:12.381190 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:47:12.384504 (+ 3314us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:47:12.384511 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:47:12.384526 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:47:12.384561 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:47:12.386789 (+ 2228us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:47:12.387433 (+ 644us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:47:12.387440 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:47:12.387449 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:47:12.387640 (+ 191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:47:12.388079 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:47:12.388082 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:47:12.388084 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:47:12.388772 (+ 688us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:47:12.389675 (+ 903us) server_negotiation.cc:1035] Waiting for connection context
0925 22:47:12.389880 (+ 205us) server_negotiation.cc:299] Negotiation successful
0925 22:47:12.390287 (+ 407us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":309,"thread_start_us":164,"threads_started":1}
I20260925 22:47:12.394598 24496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:47:12.383447 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.198.60:39445 (local address 127.0.0.1:37734)
0925 22:47:12.383639 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0925 22:47:12.383656 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 22:47:12.383752 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:47:12.384161 (+ 409us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:47:12.384165 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:47:12.384345 (+ 180us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:47:12.384474 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:47:12.384480 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:47:12.385471 (+ 991us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:47:12.385476 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:47:12.386283 (+ 807us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:47:12.386295 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:47:12.387354 (+ 1059us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:47:12.387361 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:47:12.387477 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:47:12.388083 (+ 606us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:47:12.388100 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:47:12.389707 (+ 1607us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:47:12.392699 (+ 2992us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:47:12.392704 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:47:12.392706 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:47:12.392988 (+ 282us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:47:12.393376 (+ 388us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:47:12.393380 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:47:12.393382 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:47:12.393503 (+ 121us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:47:12.394025 (+ 522us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:47:12.394029 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:47:12.394150 (+ 121us) client_negotiation.cc:769] Sending connection context
0925 22:47:12.394242 (+ 92us) client_negotiation.cc:240] Negotiation successful
0925 22:47:12.394426 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117}
I20260925 22:47:12.395601 24572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:47:12.383517 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37734 (local address 127.10.198.60:39445)
0925 22:47:12.383900 (+ 383us) server_negotiation.cc:206] Beginning negotiation
0925 22:47:12.383909 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 22:47:12.383921 (+ 12us) server_negotiation.cc:407] Connection header received
0925 22:47:12.383957 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:47:12.383959 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:47:12.384000 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:47:12.384097 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:47:12.384594 (+ 497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:47:12.385349 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:47:12.386466 (+ 1117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:47:12.387192 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:47:12.387739 (+ 547us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:47:12.389868 (+ 2129us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:47:12.389871 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:47:12.389874 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:47:12.389894 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:47:12.392542 (+ 2648us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:47:12.393133 (+ 591us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:47:12.393139 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:47:12.393141 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:47:12.393235 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:47:12.393652 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:47:12.393656 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:47:12.393657 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:47:12.393894 (+ 237us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:47:12.395112 (+ 1218us) server_negotiation.cc:1035] Waiting for connection context
0925 22:47:12.395202 (+ 90us) server_negotiation.cc:299] Negotiation successful
0925 22:47:12.395349 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":89,"server-negotiator.queue_time_us":281,"thread_start_us":163,"threads_started":1}
I20260925 22:47:12.397080 24420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:47:12.382522 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37916 (local address 127.10.198.62:36499)
0925 22:47:12.382693 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0925 22:47:12.382698 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:47:12.383490 (+ 792us) server_negotiation.cc:407] Connection header received
0925 22:47:12.383743 (+ 253us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:47:12.383749 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:47:12.383811 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:47:12.383894 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:47:12.385039 (+ 1145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:47:12.385866 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:47:12.387347 (+ 1481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:47:12.388128 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:47:12.388235 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:47:12.392202 (+ 3967us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:47:12.392206 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:47:12.392209 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:47:12.392250 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:47:12.394159 (+ 1909us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:47:12.395278 (+ 1119us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:47:12.395282 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:47:12.395284 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:47:12.395387 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:47:12.395774 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:47:12.395778 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:47:12.395780 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:47:12.396077 (+ 297us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:47:12.396166 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0925 22:47:12.396717 (+ 551us) server_negotiation.cc:299] Negotiation successful
0925 22:47:12.396864 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260925 22:47:12.397123 24571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:47:12.382425 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.198.62:36499 (local address 127.0.0.1:37916)
0925 22:47:12.383320 (+ 895us) negotiation.cc:107] Waiting for socket to connect
0925 22:47:12.383405 (+ 85us) client_negotiation.cc:174] Beginning negotiation
0925 22:47:12.383606 (+ 201us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:47:12.384131 (+ 525us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:47:12.384139 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:47:12.384689 (+ 550us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:47:12.384889 (+ 200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:47:12.384907 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:47:12.386005 (+ 1098us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:47:12.386008 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:47:12.387170 (+ 1162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:47:12.387184 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:47:12.388447 (+ 1263us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:47:12.388452 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:47:12.388608 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:47:12.389198 (+ 590us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:47:12.389214 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:47:12.391972 (+ 2758us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:47:12.394763 (+ 2791us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:47:12.394776 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:47:12.394784 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:47:12.395145 (+ 361us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:47:12.395533 (+ 388us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:47:12.395538 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:47:12.395541 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:47:12.395643 (+ 102us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:47:12.396262 (+ 619us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:47:12.396271 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:47:12.396427 (+ 156us) client_negotiation.cc:769] Sending connection context
0925 22:47:12.396645 (+ 218us) client_negotiation.cc:240] Negotiation successful
0925 22:47:12.396893 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":685,"mutex_wait_us":55,"thread_start_us":127,"threads_started":1}
I20260925 22:47:12.398335 24567 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.198.61" port: 40537 } has no permanent_uuid. Determining permanent_uuid...
I20260925 22:47:12.399435 24492 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cabf5b6801764facb094d08b9cd902db: Bootstrap starting.
I20260925 22:47:12.402868 24492 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cabf5b6801764facb094d08b9cd902db: Neither blocks nor log segments found. Creating new log.
Sep 25 22:47:12 dist-test-slave-5vdr krb5kdc[24336](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376431, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.198.62@KRBTEST.COM
I20260925 22:47:12.403957 24492 log.cc:824] T 00000000000000000000000000000000 P cabf5b6801764facb094d08b9cd902db: Log is configured to *not* fsync() on all Append() calls
Sep 25 22:47:12 dist-test-slave-5vdr krb5kdc[24336](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376432, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.198.60@KRBTEST.COM for kudu/127.10.198.61@KRBTEST.COM
I20260925 22:47:12.406881 24492 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cabf5b6801764facb094d08b9cd902db: No bootstrap required, opened a new log
I20260925 22:47:12.409744 24420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:47:12.395270 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37922 (local address 127.10.198.62:36499)
0925 22:47:12.397258 (+ 1988us) server_negotiation.cc:206] Beginning negotiation
0925 22:47:12.397263 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:47:12.397277 (+ 14us) server_negotiation.cc:407] Connection header received
0925 22:47:12.397313 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:47:12.397316 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:47:12.397357 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:47:12.397437 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:47:12.398811 (+ 1374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:47:12.399543 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:47:12.400786 (+ 1243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:47:12.401181 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:47:12.401290 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:47:12.404525 (+ 3235us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:47:12.404530 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:47:12.404533 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:47:12.404564 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:47:12.406276 (+ 1712us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:47:12.406928 (+ 652us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:47:12.406936 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:47:12.406939 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:47:12.407059 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:47:12.407773 (+ 714us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:47:12.407779 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:47:12.407781 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:47:12.408396 (+ 615us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:47:12.408677 (+ 281us) server_negotiation.cc:1035] Waiting for connection context
0925 22:47:12.409479 (+ 802us) server_negotiation.cc:299] Negotiation successful
0925 22:47:12.409611 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1925,"thread_start_us":86,"threads_started":1}
I20260925 22:47:12.411388 24494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:47:12.399189 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52496 (local address 127.10.198.61:40537)
0925 22:47:12.399303 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0925 22:47:12.399308 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:47:12.399564 (+ 256us) server_negotiation.cc:407] Connection header received
0925 22:47:12.399696 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:47:12.399700 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:47:12.399749 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:47:12.399826 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:47:12.400472 (+ 646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:47:12.401347 (+ 875us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:47:12.403101 (+ 1754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:47:12.403529 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:47:12.403629 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:47:12.407014 (+ 3385us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:47:12.407019 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:47:12.407022 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:47:12.407050 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:47:12.408811 (+ 1761us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:47:12.409381 (+ 570us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:47:12.409386 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:47:12.409388 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:47:12.409481 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:47:12.409910 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:47:12.409915 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:47:12.409916 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:47:12.410638 (+ 722us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:47:12.410765 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0925 22:47:12.411102 (+ 337us) server_negotiation.cc:299] Negotiation successful
0925 22:47:12.411242 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260925 22:47:12.411758 24492 raft_consensus.cc:348] T 00000000000000000000000000000000 P cabf5b6801764facb094d08b9cd902db [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2bffee41198446adab3908db924fa468" member_type: VOTER last_known_addr { host: "127.10.198.62" port: 36499 } } peers { permanent_uuid: "cabf5b6801764facb094d08b9cd902db" member_type: VOTER last_known_addr { host: "127.10.198.61" port: 40537 } } peers { permanent_uuid: "c296129fe03540e89de350f61469a203" member_type: VOTER last_known_addr { host: "127.10.198.60" port: 39445 } }
I20260925 22:47:12.412065 24492 raft_consensus.cc:374] T 00000000000000000000000000000000 P cabf5b6801764facb094d08b9cd902db [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:47:12.412143 24492 raft_consensus.cc:729] T 00000000000000000000000000000000 P cabf5b6801764facb094d08b9cd902db [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cabf5b6801764facb094d08b9cd902db, State: Initialized, Role: FOLLOWER
I20260925 22:47:12.412756 24492 consensus_queue.cc:260] T 00000000000000000000000000000000 P cabf5b6801764facb094d08b9cd902db [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2bffee41198446adab3908db924fa468" member_type: VOTER last_known_addr { host: "127.10.198.62" port: 36499 } } peers { permanent_uuid: "cabf5b6801764facb094d08b9cd902db" member_type: VOTER last_known_addr { host: "127.10.198.61" port: 40537 } } peers { permanent_uuid: "c296129fe03540e89de350f61469a203" member_type: VOTER last_known_addr { host: "127.10.198.60" port: 39445 } }
I20260925 22:47:12.413309 24576 sys_catalog.cc:455] T 00000000000000000000000000000000 P cabf5b6801764facb094d08b9cd902db [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2bffee41198446adab3908db924fa468" member_type: VOTER last_known_addr { host: "127.10.198.62" port: 36499 } } peers { permanent_uuid: "cabf5b6801764facb094d08b9cd902db" member_type: VOTER last_known_addr { host: "127.10.198.61" port: 40537 } } peers { permanent_uuid: "c296129fe03540e89de350f61469a203" member_type: VOTER last_known_addr { host: "127.10.198.60" port: 39445 } } }
I20260925 22:47:12.413481 24576 sys_catalog.cc:458] T 00000000000000000000000000000000 P cabf5b6801764facb094d08b9cd902db [sys.catalog]: This master's current role is: FOLLOWER
I20260925 22:47:12.414314 24571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:47:12.398911 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.198.61:40537 (local address 127.0.0.1:52496)
0925 22:47:12.399477 (+ 566us) negotiation.cc:107] Waiting for socket to connect
0925 22:47:12.399494 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 22:47:12.399604 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:47:12.399938 (+ 334us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:47:12.399941 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:47:12.400169 (+ 228us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:47:12.400319 (+ 150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:47:12.400326 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:47:12.401525 (+ 1199us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:47:12.401531 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:47:12.402923 (+ 1392us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:47:12.402943 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:47:12.403862 (+ 919us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:47:12.403869 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:47:12.403988 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:47:12.404566 (+ 578us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:47:12.404585 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:47:12.406806 (+ 2221us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:47:12.408974 (+ 2168us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:47:12.408978 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:47:12.408980 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:47:12.409258 (+ 278us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:47:12.409603 (+ 345us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:47:12.409606 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:47:12.409610 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:47:12.409704 (+ 94us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:47:12.410753 (+ 1049us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:47:12.410756 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:47:12.410880 (+ 124us) client_negotiation.cc:769] Sending connection context
0925 22:47:12.413940 (+ 3060us) client_negotiation.cc:240] Negotiation successful
0925 22:47:12.414142 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":477}
I20260925 22:47:12.414390 24492 sys_catalog.cc:564] T 00000000000000000000000000000000 P cabf5b6801764facb094d08b9cd902db [sys.catalog]: configured and running, proceeding with master startup.
I20260925 22:47:12.415410 24567 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.198.60" port: 39445 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 22:47:12 dist-test-slave-5vdr krb5kdc[24336](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376431, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.198.61@KRBTEST.COM
W20260925 22:47:12.420879 24587 catalog_manager.cc:1558] T 00000000000000000000000000000000 P cabf5b6801764facb094d08b9cd902db: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 22:47:12.421089 24587 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
Sep 25 22:47:12 dist-test-slave-5vdr krb5kdc[24336](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376432, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.198.60@KRBTEST.COM for kudu/127.10.198.60@KRBTEST.COM
I20260925 22:47:12.419231 24424 master_runner.cc:186] Error getting master registration for 127.10.198.62:36499: OK, instance_id { permanent_uuid: "2bffee41198446adab3908db924fa468" instance_seqno: 1790376431881795 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260925 22:47:12.424172 24424 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260925 22:47:12.423972 24494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:47:12.411270 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52498 (local address 127.10.198.61:40537)
0925 22:47:12.411940 (+ 670us) server_negotiation.cc:206] Beginning negotiation
0925 22:47:12.411944 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:47:12.411958 (+ 14us) server_negotiation.cc:407] Connection header received
0925 22:47:12.411994 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:47:12.411997 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:47:12.412043 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:47:12.412127 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:47:12.413061 (+ 934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:47:12.413776 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:47:12.414829 (+ 1053us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:47:12.415264 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:47:12.415416 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:47:12.419201 (+ 3785us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:47:12.419207 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:47:12.419214 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:47:12.419249 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:47:12.421405 (+ 2156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:47:12.422021 (+ 616us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:47:12.422025 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:47:12.422027 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:47:12.422123 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:47:12.422421 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:47:12.422424 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:47:12.422426 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:47:12.422663 (+ 237us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:47:12.422826 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0925 22:47:12.423619 (+ 793us) server_negotiation.cc:299] Negotiation successful
0925 22:47:12.423758 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":596,"thread_start_us":143,"threads_started":1}
I20260925 22:47:12.426625 24571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:47:12.416359 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.198.60:39445 (local address 127.0.0.1:37736)
0925 22:47:12.416503 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0925 22:47:12.416518 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 22:47:12.416613 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:47:12.417006 (+ 393us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:47:12.417008 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:47:12.417219 (+ 211us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:47:12.417316 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:47:12.417323 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:47:12.418292 (+ 969us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:47:12.418295 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:47:12.419049 (+ 754us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:47:12.419061 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:47:12.420085 (+ 1024us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:47:12.420089 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:47:12.420168 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:47:12.420839 (+ 671us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:47:12.420854 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:47:12.422558 (+ 1704us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:47:12.424708 (+ 2150us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:47:12.424711 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:47:12.424713 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:47:12.425000 (+ 287us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:47:12.425317 (+ 317us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:47:12.425320 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:47:12.425324 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:47:12.425451 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:47:12.426022 (+ 571us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:47:12.426025 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:47:12.426143 (+ 118us) client_negotiation.cc:769] Sending connection context
0925 22:47:12.426264 (+ 121us) client_negotiation.cc:240] Negotiation successful
0925 22:47:12.426439 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":65}
I20260925 22:47:12.427240 24572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:47:12.416548 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37736 (local address 127.10.198.60:39445)
0925 22:47:12.416651 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0925 22:47:12.416657 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 22:47:12.416670 (+ 13us) server_negotiation.cc:407] Connection header received
0925 22:47:12.416703 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:47:12.416706 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:47:12.416739 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:47:12.416812 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:47:12.417449 (+ 637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:47:12.418172 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:47:12.419176 (+ 1004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:47:12.419499 (+ 323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:47:12.419958 (+ 459us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:47:12.422717 (+ 2759us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:47:12.422721 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:47:12.422724 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:47:12.422749 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:47:12.424565 (+ 1816us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:47:12.425117 (+ 552us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:47:12.425120 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:47:12.425122 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:47:12.425204 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:47:12.425571 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:47:12.425575 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:47:12.425576 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:47:12.425877 (+ 301us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:47:12.426001 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0925 22:47:12.426941 (+ 940us) server_negotiation.cc:299] Negotiation successful
0925 22:47:12.427086 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260925 22:47:12.428507 11032 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 22:47:12.430892 24567 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c296129fe03540e89de350f61469a203: Bootstrap starting.
I20260925 22:47:12.434365 24567 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c296129fe03540e89de350f61469a203: Neither blocks nor log segments found. Creating new log.
I20260925 22:47:12.435544 24567 log.cc:824] T 00000000000000000000000000000000 P c296129fe03540e89de350f61469a203: Log is configured to *not* fsync() on all Append() calls
I20260925 22:47:12.442886 24567 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c296129fe03540e89de350f61469a203: No bootstrap required, opened a new log
I20260925 22:47:12.443632 24420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:47:12.430731 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37934 (local address 127.10.198.62:36499)
0925 22:47:12.430863 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0925 22:47:12.430867 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:47:12.432136 (+ 1269us) server_negotiation.cc:407] Connection header received
0925 22:47:12.432317 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:47:12.432321 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:47:12.432383 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:47:12.432468 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:47:12.433392 (+ 924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:47:12.434120 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:47:12.435203 (+ 1083us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:47:12.435553 (+ 350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:47:12.435639 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:47:12.438391 (+ 2752us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:47:12.438395 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:47:12.438398 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:47:12.438446 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:47:12.440573 (+ 2127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:47:12.442032 (+ 1459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:47:12.442039 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:47:12.442041 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:47:12.442152 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:47:12.442518 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:47:12.442522 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:47:12.442524 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:47:12.442780 (+ 256us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:47:12.442891 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0925 22:47:12.443303 (+ 412us) server_negotiation.cc:299] Negotiation successful
0925 22:47:12.443471 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260925 22:47:12.447019 24494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:47:12.431473 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52510 (local address 127.10.198.61:40537)
0925 22:47:12.431593 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0925 22:47:12.431599 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 22:47:12.436559 (+ 4960us) server_negotiation.cc:407] Connection header received
0925 22:47:12.436780 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:47:12.436783 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:47:12.436847 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:47:12.436940 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:47:12.438135 (+ 1195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:47:12.438926 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:47:12.440223 (+ 1297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:47:12.440591 (+ 368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:47:12.440706 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:47:12.442645 (+ 1939us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:47:12.442656 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:47:12.442659 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:47:12.442693 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:47:12.444482 (+ 1789us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:47:12.445098 (+ 616us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:47:12.445104 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:47:12.445108 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:47:12.445214 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:47:12.445539 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:47:12.445546 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:47:12.445548 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:47:12.445841 (+ 293us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:47:12.446003 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0925 22:47:12.446689 (+ 686us) server_negotiation.cc:299] Negotiation successful
0925 22:47:12.446831 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260925 22:47:12.447276 24572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:47:12.430728 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37742 (local address 127.10.198.60:39445)
0925 22:47:12.431240 (+ 512us) server_negotiation.cc:206] Beginning negotiation
0925 22:47:12.431247 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 22:47:12.431479 (+ 232us) server_negotiation.cc:407] Connection header received
0925 22:47:12.431688 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:47:12.431691 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:47:12.431746 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:47:12.431812 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:47:12.432988 (+ 1176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:47:12.433794 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:47:12.435251 (+ 1457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:47:12.435579 (+ 328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:47:12.435649 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:47:12.438406 (+ 2757us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:47:12.438411 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:47:12.438413 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:47:12.438448 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:47:12.441384 (+ 2936us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:47:12.445887 (+ 4503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:47:12.445891 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:47:12.445893 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:47:12.446006 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:47:12.446310 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:47:12.446313 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:47:12.446315 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:47:12.446536 (+ 221us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:47:12.446649 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0925 22:47:12.446988 (+ 339us) server_negotiation.cc:299] Negotiation successful
0925 22:47:12.447120 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":416}
I20260925 22:47:12.449422 24567 raft_consensus.cc:348] T 00000000000000000000000000000000 P c296129fe03540e89de350f61469a203 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2bffee41198446adab3908db924fa468" member_type: VOTER last_known_addr { host: "127.10.198.62" port: 36499 } } peers { permanent_uuid: "cabf5b6801764facb094d08b9cd902db" member_type: VOTER last_known_addr { host: "127.10.198.61" port: 40537 } } peers { permanent_uuid: "c296129fe03540e89de350f61469a203" member_type: VOTER last_known_addr { host: "127.10.198.60" port: 39445 } }
I20260925 22:47:12.449693 24567 raft_consensus.cc:374] T 00000000000000000000000000000000 P c296129fe03540e89de350f61469a203 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:47:12.449730 24567 raft_consensus.cc:729] T 00000000000000000000000000000000 P c296129fe03540e89de350f61469a203 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c296129fe03540e89de350f61469a203, State: Initialized, Role: FOLLOWER
I20260925 22:47:12.450232 24567 consensus_queue.cc:260] T 00000000000000000000000000000000 P c296129fe03540e89de350f61469a203 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2bffee41198446adab3908db924fa468" member_type: VOTER last_known_addr { host: "127.10.198.62" port: 36499 } } peers { permanent_uuid: "cabf5b6801764facb094d08b9cd902db" member_type: VOTER last_known_addr { host: "127.10.198.61" port: 40537 } } peers { permanent_uuid: "c296129fe03540e89de350f61469a203" member_type: VOTER last_known_addr { host: "127.10.198.60" port: 39445 } }
I20260925 22:47:12.450800 24597 sys_catalog.cc:455] T 00000000000000000000000000000000 P c296129fe03540e89de350f61469a203 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2bffee41198446adab3908db924fa468" member_type: VOTER last_known_addr { host: "127.10.198.62" port: 36499 } } peers { permanent_uuid: "cabf5b6801764facb094d08b9cd902db" member_type: VOTER last_known_addr { host: "127.10.198.61" port: 40537 } } peers { permanent_uuid: "c296129fe03540e89de350f61469a203" member_type: VOTER last_known_addr { host: "127.10.198.60" port: 39445 } } }
I20260925 22:47:12.450954 24597 sys_catalog.cc:458] T 00000000000000000000000000000000 P c296129fe03540e89de350f61469a203 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 22:47:12.451769 24567 sys_catalog.cc:564] T 00000000000000000000000000000000 P c296129fe03540e89de350f61469a203 [sys.catalog]: configured and running, proceeding with master startup.
W20260925 22:47:12.459285 24608 catalog_manager.cc:1558] T 00000000000000000000000000000000 P c296129fe03540e89de350f61469a203: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 22:47:12.459729 24608 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 22:47:12.457063 24498 master_runner.cc:186] Error getting master registration for 127.10.198.62:36499: OK, instance_id { permanent_uuid: "2bffee41198446adab3908db924fa468" instance_seqno: 1790376431881795 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260925 22:47:12.460819 24498 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260925 22:47:12.481163 24576 raft_consensus.cc:482] T 00000000000000000000000000000000 P cabf5b6801764facb094d08b9cd902db [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 22:47:12.481443 24576 raft_consensus.cc:504] T 00000000000000000000000000000000 P cabf5b6801764facb094d08b9cd902db [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2bffee41198446adab3908db924fa468" member_type: VOTER last_known_addr { host: "127.10.198.62" port: 36499 } } peers { permanent_uuid: "cabf5b6801764facb094d08b9cd902db" member_type: VOTER last_known_addr { host: "127.10.198.61" port: 40537 } } peers { permanent_uuid: "c296129fe03540e89de350f61469a203" member_type: VOTER last_known_addr { host: "127.10.198.60" port: 39445 } }
I20260925 22:47:12.482604 24576 leader_election.cc:288] T 00000000000000000000000000000000 P cabf5b6801764facb094d08b9cd902db [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2bffee41198446adab3908db924fa468 (127.10.198.62:36499), c296129fe03540e89de350f61469a203 (127.10.198.60:39445)
I20260925 22:47:12.483361 24541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cabf5b6801764facb094d08b9cd902db" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c296129fe03540e89de350f61469a203" is_pre_election: true
I20260925 22:47:12.483843 24541 raft_consensus.cc:2454] T 00000000000000000000000000000000 P c296129fe03540e89de350f61469a203 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cabf5b6801764facb094d08b9cd902db in term 0.
I20260925 22:47:12.484405 24440 leader_election.cc:302] T 00000000000000000000000000000000 P cabf5b6801764facb094d08b9cd902db [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c296129fe03540e89de350f61469a203, cabf5b6801764facb094d08b9cd902db; no voters:
I20260925 22:47:12.484761 24576 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cabf5b6801764facb094d08b9cd902db [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 22:47:12.485028 24576 raft_consensus.cc:482] T 00000000000000000000000000000000 P cabf5b6801764facb094d08b9cd902db [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 22:47:12.485126 24576 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cabf5b6801764facb094d08b9cd902db [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:47:12.486822 24576 raft_consensus.cc:504] T 00000000000000000000000000000000 P cabf5b6801764facb094d08b9cd902db [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2bffee41198446adab3908db924fa468" member_type: VOTER last_known_addr { host: "127.10.198.62" port: 36499 } } peers { permanent_uuid: "cabf5b6801764facb094d08b9cd902db" member_type: VOTER last_known_addr { host: "127.10.198.61" port: 40537 } } peers { permanent_uuid: "c296129fe03540e89de350f61469a203" member_type: VOTER last_known_addr { host: "127.10.198.60" port: 39445 } }
I20260925 22:47:12.487285 24576 leader_election.cc:288] T 00000000000000000000000000000000 P cabf5b6801764facb094d08b9cd902db [CANDIDATE]: Term 1 election: Requested vote from peers 2bffee41198446adab3908db924fa468 (127.10.198.62:36499), c296129fe03540e89de350f61469a203 (127.10.198.60:39445)
I20260925 22:47:12.487978 24541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cabf5b6801764facb094d08b9cd902db" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c296129fe03540e89de350f61469a203"
I20260925 22:47:12.488135 24541 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c296129fe03540e89de350f61469a203 [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:47:12.489768 24541 raft_consensus.cc:2454] T 00000000000000000000000000000000 P c296129fe03540e89de350f61469a203 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cabf5b6801764facb094d08b9cd902db in term 1.
I20260925 22:47:12.490147 24440 leader_election.cc:302] T 00000000000000000000000000000000 P cabf5b6801764facb094d08b9cd902db [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c296129fe03540e89de350f61469a203, cabf5b6801764facb094d08b9cd902db; no voters:
I20260925 22:47:12.490372 24576 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cabf5b6801764facb094d08b9cd902db [term 1 FOLLOWER]: Leader election won for term 1
I20260925 22:47:12.490772 24576 raft_consensus.cc:686] T 00000000000000000000000000000000 P cabf5b6801764facb094d08b9cd902db [term 1 LEADER]: Becoming Leader. State: Replica: cabf5b6801764facb094d08b9cd902db, State: Running, Role: LEADER
I20260925 22:47:12.491365 24576 consensus_queue.cc:237] T 00000000000000000000000000000000 P cabf5b6801764facb094d08b9cd902db [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2bffee41198446adab3908db924fa468" member_type: VOTER last_known_addr { host: "127.10.198.62" port: 36499 } } peers { permanent_uuid: "cabf5b6801764facb094d08b9cd902db" member_type: VOTER last_known_addr { host: "127.10.198.61" port: 40537 } } peers { permanent_uuid: "c296129fe03540e89de350f61469a203" member_type: VOTER last_known_addr { host: "127.10.198.60" port: 39445 } }
I20260925 22:47:12.493929 24610 sys_catalog.cc:455] T 00000000000000000000000000000000 P cabf5b6801764facb094d08b9cd902db [sys.catalog]: SysCatalogTable state changed. Reason: New leader cabf5b6801764facb094d08b9cd902db. Latest consensus state: current_term: 1 leader_uuid: "cabf5b6801764facb094d08b9cd902db" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2bffee41198446adab3908db924fa468" member_type: VOTER last_known_addr { host: "127.10.198.62" port: 36499 } } peers { permanent_uuid: "cabf5b6801764facb094d08b9cd902db" member_type: VOTER last_known_addr { host: "127.10.198.61" port: 40537 } } peers { permanent_uuid: "c296129fe03540e89de350f61469a203" member_type: VOTER last_known_addr { host: "127.10.198.60" port: 39445 } } }
I20260925 22:47:12.494114 24610 sys_catalog.cc:458] T 00000000000000000000000000000000 P cabf5b6801764facb094d08b9cd902db [sys.catalog]: This master's current role is: LEADER
I20260925 22:47:12.494704 24612 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 22:47:12.495939 24612 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 22:47:12.503655 24541 raft_consensus.cc:1261] T 00000000000000000000000000000000 P c296129fe03540e89de350f61469a203 [term 1 FOLLOWER]: Refusing update from remote peer cabf5b6801764facb094d08b9cd902db: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 22:47:12.504546 24610 consensus_queue.cc:1035] T 00000000000000000000000000000000 P cabf5b6801764facb094d08b9cd902db [LEADER]: Connected to new peer: Peer: permanent_uuid: "c296129fe03540e89de350f61469a203" member_type: VOTER last_known_addr { host: "127.10.198.60" port: 39445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 22:47:12.513772 24597 sys_catalog.cc:455] T 00000000000000000000000000000000 P c296129fe03540e89de350f61469a203 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cabf5b6801764facb094d08b9cd902db. Latest consensus state: current_term: 1 leader_uuid: "cabf5b6801764facb094d08b9cd902db" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2bffee41198446adab3908db924fa468" member_type: VOTER last_known_addr { host: "127.10.198.62" port: 36499 } } peers { permanent_uuid: "cabf5b6801764facb094d08b9cd902db" member_type: VOTER last_known_addr { host: "127.10.198.61" port: 40537 } } peers { permanent_uuid: "c296129fe03540e89de350f61469a203" member_type: VOTER last_known_addr { host: "127.10.198.60" port: 39445 } } }
I20260925 22:47:12.513978 24597 sys_catalog.cc:458] T 00000000000000000000000000000000 P c296129fe03540e89de350f61469a203 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 22:47:12.518193 24576 sys_catalog.cc:455] T 00000000000000000000000000000000 P cabf5b6801764facb094d08b9cd902db [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "cabf5b6801764facb094d08b9cd902db" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2bffee41198446adab3908db924fa468" member_type: VOTER last_known_addr { host: "127.10.198.62" port: 36499 } } peers { permanent_uuid: "cabf5b6801764facb094d08b9cd902db" member_type: VOTER last_known_addr { host: "127.10.198.61" port: 40537 } } peers { permanent_uuid: "c296129fe03540e89de350f61469a203" member_type: VOTER last_known_addr { host: "127.10.198.60" port: 39445 } } }
I20260925 22:47:12.518376 24576 sys_catalog.cc:458] T 00000000000000000000000000000000 P cabf5b6801764facb094d08b9cd902db [sys.catalog]: This master's current role is: LEADER
I20260925 22:47:12.518517 24613 mvcc.cc:205] Tried to move back new op lower bound from 7333381867524485120 to 7333381867488391168. Current Snapshot: MvccSnapshot[applied={T|T < 7333381867524485120}]
I20260925 22:47:12.520078 24597 sys_catalog.cc:455] T 00000000000000000000000000000000 P c296129fe03540e89de350f61469a203 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "cabf5b6801764facb094d08b9cd902db" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2bffee41198446adab3908db924fa468" member_type: VOTER last_known_addr { host: "127.10.198.62" port: 36499 } } peers { permanent_uuid: "cabf5b6801764facb094d08b9cd902db" member_type: VOTER last_known_addr { host: "127.10.198.61" port: 40537 } } peers { permanent_uuid: "c296129fe03540e89de350f61469a203" member_type: VOTER last_known_addr { host: "127.10.198.60" port: 39445 } } }
I20260925 22:47:12.520241 24597 sys_catalog.cc:458] T 00000000000000000000000000000000 P c296129fe03540e89de350f61469a203 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 22:47:12.524163 24612 catalog_manager.cc:1347] Generated new cluster ID: c2e17d6fe9fc437884a6bec494ddcbec
I20260925 22:47:12.524258 24612 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 22:47:12.528475 24417 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.198.60" port: 39445 } attempt: 4
Sep 25 22:47:12 dist-test-slave-5vdr krb5kdc[24336](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376431, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.198.62@KRBTEST.COM for kudu/127.10.198.60@KRBTEST.COM
I20260925 22:47:12.541391 24612 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 22:47:12.543162 24612 catalog_manager.cc:1504] Loading token signing keys...
I20260925 22:47:12.548033 24572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:47:12.529675 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37744 (local address 127.10.198.60:39445)
0925 22:47:12.529783 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0925 22:47:12.529787 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:47:12.529798 (+ 11us) server_negotiation.cc:407] Connection header received
0925 22:47:12.529854 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:47:12.529857 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:47:12.529920 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:47:12.529984 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:47:12.530719 (+ 735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:47:12.531510 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:47:12.532760 (+ 1250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:47:12.533105 (+ 345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:47:12.533204 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:47:12.537720 (+ 4516us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:47:12.537724 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:47:12.537727 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:47:12.537759 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:47:12.542649 (+ 4890us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:47:12.543987 (+ 1338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:47:12.544027 (+ 40us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:47:12.544029 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:47:12.544260 (+ 231us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:47:12.545019 (+ 759us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:47:12.545022 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:47:12.545024 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:47:12.545552 (+ 528us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:47:12.546168 (+ 616us) server_negotiation.cc:1035] Waiting for connection context
0925 22:47:12.546692 (+ 524us) server_negotiation.cc:299] Negotiation successful
0925 22:47:12.547299 (+ 607us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260925 22:47:12.548101 24421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:47:12.529141 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.198.60:39445 (local address 127.0.0.1:37744)
0925 22:47:12.529426 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0925 22:47:12.529441 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 22:47:12.529532 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:47:12.530170 (+ 638us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:47:12.530173 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:47:12.530407 (+ 234us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:47:12.530509 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:47:12.530515 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:47:12.531701 (+ 1186us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:47:12.531705 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:47:12.532561 (+ 856us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:47:12.532571 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:47:12.533401 (+ 830us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:47:12.533404 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:47:12.533496 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:47:12.534113 (+ 617us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:47:12.534128 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:47:12.537489 (+ 3361us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:47:12.543070 (+ 5581us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:47:12.543075 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:47:12.543077 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:47:12.543752 (+ 675us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:47:12.544549 (+ 797us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:47:12.544553 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:47:12.544555 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:47:12.544805 (+ 250us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:47:12.545701 (+ 896us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:47:12.545704 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:47:12.546292 (+ 588us) client_negotiation.cc:769] Sending connection context
0925 22:47:12.546818 (+ 526us) client_negotiation.cc:240] Negotiation successful
0925 22:47:12.547378 (+ 560us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":199}
I20260925 22:47:12.552109 24417 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2bffee41198446adab3908db924fa468: Bootstrap starting.
I20260925 22:47:12.558614 24612 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cabf5b6801764facb094d08b9cd902db: Generated new TSK 0
I20260925 22:47:12.560494 24612 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 22:47:12.563923 24417 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2bffee41198446adab3908db924fa468: Neither blocks nor log segments found. Creating new log.
I20260925 22:47:12.565049 24417 log.cc:824] T 00000000000000000000000000000000 P 2bffee41198446adab3908db924fa468: Log is configured to *not* fsync() on all Append() calls
I20260925 22:47:12.568919 24417 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2bffee41198446adab3908db924fa468: No bootstrap required, opened a new log
I20260925 22:47:12.575213 24417 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2bffee41198446adab3908db924fa468 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2bffee41198446adab3908db924fa468" member_type: VOTER last_known_addr { host: "127.10.198.62" port: 36499 } } peers { permanent_uuid: "cabf5b6801764facb094d08b9cd902db" member_type: VOTER last_known_addr { host: "127.10.198.61" port: 40537 } } peers { permanent_uuid: "c296129fe03540e89de350f61469a203" member_type: VOTER last_known_addr { host: "127.10.198.60" port: 39445 } }
I20260925 22:47:12.575547 24417 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2bffee41198446adab3908db924fa468 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:47:12.575624 24417 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2bffee41198446adab3908db924fa468 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2bffee41198446adab3908db924fa468, State: Initialized, Role: FOLLOWER
I20260925 22:47:12.576248 24417 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2bffee41198446adab3908db924fa468 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2bffee41198446adab3908db924fa468" member_type: VOTER last_known_addr { host: "127.10.198.62" port: 36499 } } peers { permanent_uuid: "cabf5b6801764facb094d08b9cd902db" member_type: VOTER last_known_addr { host: "127.10.198.61" port: 40537 } } peers { permanent_uuid: "c296129fe03540e89de350f61469a203" member_type: VOTER last_known_addr { host: "127.10.198.60" port: 39445 } }
I20260925 22:47:12.577984 24417 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2bffee41198446adab3908db924fa468 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 22:47:12.578927 24389 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2bffee41198446adab3908db924fa468 [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:47:12.580804 24390 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cabf5b6801764facb094d08b9cd902db" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2bffee41198446adab3908db924fa468"
I20260925 22:47:12.581455 24391 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cabf5b6801764facb094d08b9cd902db" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2bffee41198446adab3908db924fa468" is_pre_election: true
I20260925 22:47:12.585090 24389 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 2bffee41198446adab3908db924fa468 [term 1 FOLLOWER]: Refusing update from remote peer cabf5b6801764facb094d08b9cd902db: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 22:47:12.586478 24620 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2bffee41198446adab3908db924fa468 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2bffee41198446adab3908db924fa468" member_type: VOTER last_known_addr { host: "127.10.198.62" port: 36499 } } peers { permanent_uuid: "cabf5b6801764facb094d08b9cd902db" member_type: VOTER last_known_addr { host: "127.10.198.61" port: 40537 } } peers { permanent_uuid: "c296129fe03540e89de350f61469a203" member_type: VOTER last_known_addr { host: "127.10.198.60" port: 39445 } } }
I20260925 22:47:12.586561 24576 consensus_queue.cc:1035] T 00000000000000000000000000000000 P cabf5b6801764facb094d08b9cd902db [LEADER]: Connected to new peer: Peer: permanent_uuid: "2bffee41198446adab3908db924fa468" member_type: VOTER last_known_addr { host: "127.10.198.62" port: 36499 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 22:47:12.586658 24620 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2bffee41198446adab3908db924fa468 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 22:47:12.592983 24634 mvcc.cc:205] Tried to move back new op lower bound from 7333381867742183424 to 7333381867488391168. Current Snapshot: MvccSnapshot[applied={T|T < 7333381867524485120}]
W20260925 22:47:12.597309 24632 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 2bffee41198446adab3908db924fa468: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 22:47:12.597550 24632 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 22:47:12.597792 24620 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2bffee41198446adab3908db924fa468 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cabf5b6801764facb094d08b9cd902db. Latest consensus state: current_term: 1 leader_uuid: "cabf5b6801764facb094d08b9cd902db" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2bffee41198446adab3908db924fa468" member_type: VOTER last_known_addr { host: "127.10.198.62" port: 36499 } } peers { permanent_uuid: "cabf5b6801764facb094d08b9cd902db" member_type: VOTER last_known_addr { host: "127.10.198.61" port: 40537 } } peers { permanent_uuid: "c296129fe03540e89de350f61469a203" member_type: VOTER last_known_addr { host: "127.10.198.60" port: 39445 } } }
I20260925 22:47:12.598031 24620 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2bffee41198446adab3908db924fa468 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 22:47:12.598234 24620 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2bffee41198446adab3908db924fa468 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "cabf5b6801764facb094d08b9cd902db" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2bffee41198446adab3908db924fa468" member_type: VOTER last_known_addr { host: "127.10.198.62" port: 36499 } } peers { permanent_uuid: "cabf5b6801764facb094d08b9cd902db" member_type: VOTER last_known_addr { host: "127.10.198.61" port: 40537 } } peers { permanent_uuid: "c296129fe03540e89de350f61469a203" member_type: VOTER last_known_addr { host: "127.10.198.60" port: 39445 } } }
I20260925 22:47:12.598419 24620 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2bffee41198446adab3908db924fa468 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 22:47:13.077486 24576 sys_catalog.cc:455] T 00000000000000000000000000000000 P cabf5b6801764facb094d08b9cd902db [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "cabf5b6801764facb094d08b9cd902db" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2bffee41198446adab3908db924fa468" member_type: VOTER last_known_addr { host: "127.10.198.62" port: 36499 } } peers { permanent_uuid: "cabf5b6801764facb094d08b9cd902db" member_type: VOTER last_known_addr { host: "127.10.198.61" port: 40537 } } peers { permanent_uuid: "c296129fe03540e89de350f61469a203" member_type: VOTER last_known_addr { host: "127.10.198.60" port: 39445 } } }
I20260925 22:47:13.077724 24576 sys_catalog.cc:458] T 00000000000000000000000000000000 P cabf5b6801764facb094d08b9cd902db [sys.catalog]: This master's current role is: LEADER
I20260925 22:47:13.461145 24608 catalog_manager.cc:1259] Loaded cluster ID: c2e17d6fe9fc437884a6bec494ddcbec
I20260925 22:47:13.461266 24608 catalog_manager.cc:1552] T 00000000000000000000000000000000 P c296129fe03540e89de350f61469a203: loading cluster ID for follower catalog manager: success
I20260925 22:47:13.462940 24608 catalog_manager.cc:1574] T 00000000000000000000000000000000 P c296129fe03540e89de350f61469a203: acquiring CA information for follower catalog manager: success
I20260925 22:47:13.464401 24608 catalog_manager.cc:1602] T 00000000000000000000000000000000 P c296129fe03540e89de350f61469a203: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260925 22:47:13.599013 24632 catalog_manager.cc:1259] Loaded cluster ID: c2e17d6fe9fc437884a6bec494ddcbec
I20260925 22:47:13.599145 24632 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 2bffee41198446adab3908db924fa468: loading cluster ID for follower catalog manager: success
I20260925 22:47:13.601955 24632 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 2bffee41198446adab3908db924fa468: acquiring CA information for follower catalog manager: success
I20260925 22:47:13.603919 24632 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 2bffee41198446adab3908db924fa468: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260925 22:47:13.645533 11032 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrWXm8X/build/debug/bin/kudu with pid 24349
I20260925 22:47:13.652913 11032 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrWXm8X/build/debug/bin/kudu with pid 24424
I20260925 22:47:13.659536 11032 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrWXm8X/build/debug/bin/kudu with pid 24498
2026-09-25T22:47:13Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (3153 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790376259650642-11032-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 22:47:13 dist-test-slave-5vdr krb5kdc[24647](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 22:47:13 dist-test-slave-5vdr krb5kdc[24647](info): set up 2 sockets
Sep 25 22:47:13 dist-test-slave-5vdr krb5kdc[24647](info): commencing operation
krb5kdc: starting...
W20260925 22:47:14.736315 11032 mini_kdc.cc:121] Time spent starting KDC: real 1.054s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790376259650642-11032-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790376259650642-11032-0/krb5kdc/test-user.keytab.
Sep 25 22:47:14 dist-test-slave-5vdr krb5kdc[24647](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376434, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T22:47:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T22:47:14Z Disabled control of system clock
WARNING: no policy specified for kudu/127.10.198.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.198.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.198.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790376259650642-11032-0/krb5kdc/kudu_127.10.198.62.keytab.
Entry for principal kudu/127.10.198.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790376259650642-11032-0/krb5kdc/kudu_127.10.198.62.keytab.
I20260925 22:47:14.867087 11032 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrWXm8X/build/debug/bin/kudu
/tmp/dist-test-taskrWXm8X/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790376259650642-11032-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790376259650642-11032-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790376259650642-11032-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790376259650642-11032-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.198.62:43273
--webserver_interface=127.10.198.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790376259650642-11032-0/krb5kdc/kudu_127.10.198.62.keytab
--principal=kudu/127.10.198.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.198.20:40103
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.198.62:43273
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790376259650642-11032-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790376259650642-11032-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790376259650642-11032-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:47:14.984596 24660 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:47:14.984916 24660 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:47:14.984961 24660 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:47:14.989466 24660 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 22:47:14.989555 24660 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:47:14.989576 24660 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:47:14.989594 24660 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 22:47:14.989610 24660 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 22:47:14.995579 24660 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.198.20:40103
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790376259650642-11032-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790376259650642-11032-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.198.62:43273
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790376259650642-11032-0/krb5kdc/kudu_127.10.198.62.keytab
--principal=kudu/127.10.198.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.198.62:43273
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790376259650642-11032-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.198.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24660
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790376259650642-11032-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:09:16 UTC on ccycloud.quasar-afmhyw.root.comops.site
build id 4673379
I20260925 22:47:14.996877 24660 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:47:14.998049 24660 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:47:15.007784 24668 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:47:15.007776 24666 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:47:15.007776 24665 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:47:15.008368 24660 server_base.cc:1034] running on GCE node
I20260925 22:47:15.009066 24660 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:47:15.010398 24660 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:47:15.011628 24660 hybrid_clock.cc:648] HybridClock initialized: now 1790376435011611 us; error 36 us; skew 500 ppm
Sep 25 22:47:15 dist-test-slave-5vdr krb5kdc[24647](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376435, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.198.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:47:15.014747 24660 init.cc:377] Logged in from keytab as kudu/127.10.198.62@KRBTEST.COM (short username kudu)
I20260925 22:47:15.015903 24660 webserver.cc:460] Webserver started at http://127.10.198.62:43741/ using document root <none> and password file <none>
I20260925 22:47:15.016561 24660 fs_manager.cc:359] Metadata directory not provided
I20260925 22:47:15.016669 24660 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:47:15.016939 24660 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:47:15.019291 24660 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790376259650642-11032-0/minicluster-data/master-0/data/instance:
uuid: "b769c6d60f724545b9b68839d0ec4be1"
format_stamp: "Formatted at 2026-09-25 22:47:15 on dist-test-slave-5vdr"
server_key: "e787e223dee573ca8bb3fd23943d80d2"
server_key_iv: "d04742a0f02b11f31ac4d57d0f1fc3ca"
server_key_version: "encryptionkey@0"
I20260925 22:47:15.020051 24660 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790376259650642-11032-0/minicluster-data/master-0/wal/instance:
uuid: "b769c6d60f724545b9b68839d0ec4be1"
format_stamp: "Formatted at 2026-09-25 22:47:15 on dist-test-slave-5vdr"
server_key: "e787e223dee573ca8bb3fd23943d80d2"
server_key_iv: "d04742a0f02b11f31ac4d57d0f1fc3ca"
server_key_version: "encryptionkey@0"
I20260925 22:47:15.024510 24660 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260925 22:47:15.027521 24675 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:47:15.029107 24660 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 22:47:15.029327 24660 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790376259650642-11032-0/minicluster-data/master-0/data,/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790376259650642-11032-0/minicluster-data/master-0/wal
uuid: "b769c6d60f724545b9b68839d0ec4be1"
format_stamp: "Formatted at 2026-09-25 22:47:15 on dist-test-slave-5vdr"
server_key: "e787e223dee573ca8bb3fd23943d80d2"
server_key_iv: "d04742a0f02b11f31ac4d57d0f1fc3ca"
server_key_version: "encryptionkey@0"
I20260925 22:47:15.029501 24660 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790376259650642-11032-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790376259650642-11032-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790376259650642-11032-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:47:15.061787 24660 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:47:15.065465 24660 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:47:15.065719 24660 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:47:15.073863 24660 rpc_server.cc:307] RPC server started. Bound to: 127.10.198.62:43273
I20260925 22:47:15.073901 24727 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.198.62:43273 every 8 connection(s)
I20260925 22:47:15.075014 24660 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790376259650642-11032-0/minicluster-data/master-0/data/info.pb
I20260925 22:47:15.078341 24728 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:47:15.084889 11032 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrWXm8X/build/debug/bin/kudu as pid 24660
I20260925 22:47:15.085053 11032 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790376259650642-11032-0/minicluster-data/master-0/wal/instance
I20260925 22:47:15.084998 24728 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b769c6d60f724545b9b68839d0ec4be1: Bootstrap starting.
I20260925 22:47:15.085366 11032 external_mini_cluster.cc:1428] Setting key cdadc809f4cf59e0a199d709be17aaf8
I20260925 22:47:15.088099 24728 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b769c6d60f724545b9b68839d0ec4be1: Neither blocks nor log segments found. Creating new log.
I20260925 22:47:15.089133 24728 log.cc:824] T 00000000000000000000000000000000 P b769c6d60f724545b9b68839d0ec4be1: Log is configured to *not* fsync() on all Append() calls
I20260925 22:47:15.091936 24728 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b769c6d60f724545b9b68839d0ec4be1: No bootstrap required, opened a new log
Sep 25 22:47:15 dist-test-slave-5vdr krb5kdc[24647](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376434, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.198.62@KRBTEST.COM
I20260925 22:47:15.096662 24728 raft_consensus.cc:348] T 00000000000000000000000000000000 P b769c6d60f724545b9b68839d0ec4be1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b769c6d60f724545b9b68839d0ec4be1" member_type: VOTER last_known_addr { host: "127.10.198.62" port: 43273 } }
I20260925 22:47:15.097081 24728 raft_consensus.cc:374] T 00000000000000000000000000000000 P b769c6d60f724545b9b68839d0ec4be1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:47:15.097165 24728 raft_consensus.cc:729] T 00000000000000000000000000000000 P b769c6d60f724545b9b68839d0ec4be1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b769c6d60f724545b9b68839d0ec4be1, State: Initialized, Role: FOLLOWER
I20260925 22:47:15.097754 24728 consensus_queue.cc:260] T 00000000000000000000000000000000 P b769c6d60f724545b9b68839d0ec4be1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b769c6d60f724545b9b68839d0ec4be1" member_type: VOTER last_known_addr { host: "127.10.198.62" port: 43273 } }
I20260925 22:47:15.097915 24728 raft_consensus.cc:388] T 00000000000000000000000000000000 P b769c6d60f724545b9b68839d0ec4be1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 22:47:15.098014 24728 raft_consensus.cc:482] T 00000000000000000000000000000000 P b769c6d60f724545b9b68839d0ec4be1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 22:47:15.098163 24728 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b769c6d60f724545b9b68839d0ec4be1 [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:47:15.099296 24728 raft_consensus.cc:504] T 00000000000000000000000000000000 P b769c6d60f724545b9b68839d0ec4be1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b769c6d60f724545b9b68839d0ec4be1" member_type: VOTER last_known_addr { host: "127.10.198.62" port: 43273 } }
I20260925 22:47:15.099730 24728 leader_election.cc:302] T 00000000000000000000000000000000 P b769c6d60f724545b9b68839d0ec4be1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b769c6d60f724545b9b68839d0ec4be1; no voters:
I20260925 22:47:15.100169 24728 leader_election.cc:288] T 00000000000000000000000000000000 P b769c6d60f724545b9b68839d0ec4be1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 22:47:15.100255 24733 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b769c6d60f724545b9b68839d0ec4be1 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 22:47:15.100514 24733 raft_consensus.cc:686] T 00000000000000000000000000000000 P b769c6d60f724545b9b68839d0ec4be1 [term 1 LEADER]: Becoming Leader. State: Replica: b769c6d60f724545b9b68839d0ec4be1, State: Running, Role: LEADER
I20260925 22:47:15.100857 24733 consensus_queue.cc:237] T 00000000000000000000000000000000 P b769c6d60f724545b9b68839d0ec4be1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b769c6d60f724545b9b68839d0ec4be1" member_type: VOTER last_known_addr { host: "127.10.198.62" port: 43273 } }
I20260925 22:47:15.101676 24728 sys_catalog.cc:564] T 00000000000000000000000000000000 P b769c6d60f724545b9b68839d0ec4be1 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 22:47:15.102172 24733 sys_catalog.cc:455] T 00000000000000000000000000000000 P b769c6d60f724545b9b68839d0ec4be1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b769c6d60f724545b9b68839d0ec4be1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b769c6d60f724545b9b68839d0ec4be1" member_type: VOTER last_known_addr { host: "127.10.198.62" port: 43273 } } }
I20260925 22:47:15.102304 24733 sys_catalog.cc:458] T 00000000000000000000000000000000 P b769c6d60f724545b9b68839d0ec4be1 [sys.catalog]: This master's current role is: LEADER
I20260925 22:47:15.102591 24733 sys_catalog.cc:455] T 00000000000000000000000000000000 P b769c6d60f724545b9b68839d0ec4be1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b769c6d60f724545b9b68839d0ec4be1. Latest consensus state: current_term: 1 leader_uuid: "b769c6d60f724545b9b68839d0ec4be1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b769c6d60f724545b9b68839d0ec4be1" member_type: VOTER last_known_addr { host: "127.10.198.62" port: 43273 } } }
I20260925 22:47:15.102692 24733 sys_catalog.cc:458] T 00000000000000000000000000000000 P b769c6d60f724545b9b68839d0ec4be1 [sys.catalog]: This master's current role is: LEADER
I20260925 22:47:15.103688 24737 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 22:47:15.104628 24731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:47:15.086968 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59960 (local address 127.10.198.62:43273)
0925 22:47:15.087483 (+ 515us) server_negotiation.cc:206] Beginning negotiation
0925 22:47:15.087503 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0925 22:47:15.087552 (+ 49us) server_negotiation.cc:407] Connection header received
0925 22:47:15.088627 (+ 1075us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:47:15.088656 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:47:15.089142 (+ 486us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:47:15.089586 (+ 444us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:47:15.090594 (+ 1008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:47:15.092110 (+ 1516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:47:15.093060 (+ 950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:47:15.093816 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:47:15.094013 (+ 197us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:47:15.096950 (+ 2937us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:47:15.096957 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:47:15.096978 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:47:15.097027 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:47:15.099213 (+ 2186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:47:15.099876 (+ 663us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:47:15.099886 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:47:15.099893 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:47:15.100079 (+ 186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:47:15.100510 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:47:15.100514 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:47:15.100515 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:47:15.101105 (+ 590us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:47:15.101326 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0925 22:47:15.101652 (+ 326us) server_negotiation.cc:299] Negotiation successful
0925 22:47:15.102014 (+ 362us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":342,"thread_start_us":155,"threads_started":1}
I20260925 22:47:15.107963 24737 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 22:47:15.108166 11032 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 22:47:15.116967 24737 catalog_manager.cc:1347] Generated new cluster ID: 9efe0442f5134b81af15032b9d43a66c
I20260925 22:47:15.117095 24737 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 22:47:15.125972 24737 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 22:47:15.127041 24737 catalog_manager.cc:1504] Loading token signing keys...
I20260925 22:47:15.139075 24737 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b769c6d60f724545b9b68839d0ec4be1: Generated new TSK 0
I20260925 22:47:15.140010 24737 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 22:47:15.164950 11032 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrWXm8X/build/debug/bin/kudu with pid 24660
2026-09-25T22:47:15Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (1503 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376259650642-11032-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 22:47:15 dist-test-slave-5vdr krb5kdc[24755](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 22:47:15 dist-test-slave-5vdr krb5kdc[24755](info): set up 2 sockets
Sep 25 22:47:15 dist-test-slave-5vdr krb5kdc[24755](info): commencing operation
krb5kdc: starting...
W20260925 22:47:16.220436 11032 mini_kdc.cc:121] Time spent starting KDC: real 1.035s user 0.005s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376259650642-11032-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376259650642-11032-0/krb5kdc/test-user.keytab.
Sep 25 22:47:16 dist-test-slave-5vdr krb5kdc[24755](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376436, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T22:47:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T22:47:16Z Disabled control of system clock
WARNING: no policy specified for kudu/127.10.198.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.198.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.198.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376259650642-11032-0/krb5kdc/kudu_127.10.198.62.keytab.
Entry for principal kudu/127.10.198.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376259650642-11032-0/krb5kdc/kudu_127.10.198.62.keytab.
I20260925 22:47:16.356128 11032 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrWXm8X/build/debug/bin/kudu
/tmp/dist-test-taskrWXm8X/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376259650642-11032-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376259650642-11032-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376259650642-11032-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376259650642-11032-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.198.62:42687
--webserver_interface=127.10.198.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376259650642-11032-0/krb5kdc/kudu_127.10.198.62.keytab
--principal=kudu/127.10.198.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.198.20:44287
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.198.62:42687
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376259650642-11032-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376259650642-11032-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376259650642-11032-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:47:16.483306 24768 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:47:16.483676 24768 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:47:16.483791 24768 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:47:16.488252 24768 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 22:47:16.488370 24768 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:47:16.488413 24768 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:47:16.488479 24768 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 22:47:16.488525 24768 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 22:47:16.494918 24768 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.198.20:44287
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376259650642-11032-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376259650642-11032-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.198.62:42687
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376259650642-11032-0/krb5kdc/kudu_127.10.198.62.keytab
--principal=kudu/127.10.198.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.198.62:42687
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376259650642-11032-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.198.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24768
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376259650642-11032-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:09:16 UTC on ccycloud.quasar-afmhyw.root.comops.site
build id 4673379
I20260925 22:47:16.496253 24768 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:47:16.497413 24768 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:47:16.506989 24774 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:47:16.507015 24776 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:47:16.506989 24773 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:47:16.507323 24768 server_base.cc:1034] running on GCE node
I20260925 22:47:16.508131 24768 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:47:16.509492 24768 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:47:16.510746 24768 hybrid_clock.cc:648] HybridClock initialized: now 1790376436510721 us; error 43 us; skew 500 ppm
Sep 25 22:47:16 dist-test-slave-5vdr krb5kdc[24755](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376436, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.198.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:47:16.519198 24768 init.cc:377] Logged in from keytab as kudu/127.10.198.62@KRBTEST.COM (short username kudu)
I20260925 22:47:16.521100 24768 webserver.cc:460] Webserver started at http://127.10.198.62:36809/ using document root <none> and password file <none>
I20260925 22:47:16.522305 24768 fs_manager.cc:359] Metadata directory not provided
I20260925 22:47:16.522432 24768 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:47:16.522716 24768 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:47:16.526871 24768 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376259650642-11032-0/minicluster-data/master-0/data/instance:
uuid: "27eb88732a4e4caaae541c787b446b72"
format_stamp: "Formatted at 2026-09-25 22:47:16 on dist-test-slave-5vdr"
server_key: "e87b98d930f8e15835be623fbc0bd542"
server_key_iv: "802d4f5a06d643fdbf4e2c978d2ee152"
server_key_version: "encryptionkey@0"
I20260925 22:47:16.528383 24768 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376259650642-11032-0/minicluster-data/master-0/wal/instance:
uuid: "27eb88732a4e4caaae541c787b446b72"
format_stamp: "Formatted at 2026-09-25 22:47:16 on dist-test-slave-5vdr"
server_key: "e87b98d930f8e15835be623fbc0bd542"
server_key_iv: "802d4f5a06d643fdbf4e2c978d2ee152"
server_key_version: "encryptionkey@0"
I20260925 22:47:16.535333 24768 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260925 22:47:16.539158 24783 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:47:16.541352 24768 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260925 22:47:16.541556 24768 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376259650642-11032-0/minicluster-data/master-0/data,/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376259650642-11032-0/minicluster-data/master-0/wal
uuid: "27eb88732a4e4caaae541c787b446b72"
format_stamp: "Formatted at 2026-09-25 22:47:16 on dist-test-slave-5vdr"
server_key: "e87b98d930f8e15835be623fbc0bd542"
server_key_iv: "802d4f5a06d643fdbf4e2c978d2ee152"
server_key_version: "encryptionkey@0"
I20260925 22:47:16.541761 24768 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376259650642-11032-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376259650642-11032-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376259650642-11032-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:47:16.563199 24768 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:47:16.566955 24768 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:47:16.567227 24768 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:47:16.575475 24768 rpc_server.cc:307] RPC server started. Bound to: 127.10.198.62:42687
I20260925 22:47:16.575773 24835 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.198.62:42687 every 8 connection(s)
I20260925 22:47:16.576707 24768 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376259650642-11032-0/minicluster-data/master-0/data/info.pb
I20260925 22:47:16.579928 24836 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:47:16.584553 11032 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrWXm8X/build/debug/bin/kudu as pid 24768
I20260925 22:47:16.584734 11032 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376259650642-11032-0/minicluster-data/master-0/wal/instance
I20260925 22:47:16.585019 11032 external_mini_cluster.cc:1428] Setting key c251b2f31ad2cb721f9448159621ff68
I20260925 22:47:16.587100 24836 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 27eb88732a4e4caaae541c787b446b72: Bootstrap starting.
I20260925 22:47:16.589864 24836 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 27eb88732a4e4caaae541c787b446b72: Neither blocks nor log segments found. Creating new log.
I20260925 22:47:16.590942 24836 log.cc:824] T 00000000000000000000000000000000 P 27eb88732a4e4caaae541c787b446b72: Log is configured to *not* fsync() on all Append() calls
I20260925 22:47:16.594339 24836 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 27eb88732a4e4caaae541c787b446b72: No bootstrap required, opened a new log
Sep 25 22:47:16 dist-test-slave-5vdr krb5kdc[24755](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376436, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.198.62@KRBTEST.COM
I20260925 22:47:16.599947 24836 raft_consensus.cc:348] T 00000000000000000000000000000000 P 27eb88732a4e4caaae541c787b446b72 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "27eb88732a4e4caaae541c787b446b72" member_type: VOTER last_known_addr { host: "127.10.198.62" port: 42687 } }
I20260925 22:47:16.600322 24836 raft_consensus.cc:374] T 00000000000000000000000000000000 P 27eb88732a4e4caaae541c787b446b72 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:47:16.600406 24836 raft_consensus.cc:729] T 00000000000000000000000000000000 P 27eb88732a4e4caaae541c787b446b72 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 27eb88732a4e4caaae541c787b446b72, State: Initialized, Role: FOLLOWER
I20260925 22:47:16.601017 24836 consensus_queue.cc:260] T 00000000000000000000000000000000 P 27eb88732a4e4caaae541c787b446b72 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "27eb88732a4e4caaae541c787b446b72" member_type: VOTER last_known_addr { host: "127.10.198.62" port: 42687 } }
I20260925 22:47:16.601226 24836 raft_consensus.cc:388] T 00000000000000000000000000000000 P 27eb88732a4e4caaae541c787b446b72 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 22:47:16.601320 24836 raft_consensus.cc:482] T 00000000000000000000000000000000 P 27eb88732a4e4caaae541c787b446b72 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 22:47:16.601470 24836 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 27eb88732a4e4caaae541c787b446b72 [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:47:16.602932 24836 raft_consensus.cc:504] T 00000000000000000000000000000000 P 27eb88732a4e4caaae541c787b446b72 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "27eb88732a4e4caaae541c787b446b72" member_type: VOTER last_known_addr { host: "127.10.198.62" port: 42687 } }
I20260925 22:47:16.603376 24836 leader_election.cc:302] T 00000000000000000000000000000000 P 27eb88732a4e4caaae541c787b446b72 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 27eb88732a4e4caaae541c787b446b72; no voters:
I20260925 22:47:16.603778 24836 leader_election.cc:288] T 00000000000000000000000000000000 P 27eb88732a4e4caaae541c787b446b72 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 22:47:16.604092 24841 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 27eb88732a4e4caaae541c787b446b72 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 22:47:16.604377 24839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:47:16.586624 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38596 (local address 127.10.198.62:42687)
0925 22:47:16.587155 (+ 531us) server_negotiation.cc:206] Beginning negotiation
0925 22:47:16.587172 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0925 22:47:16.587249 (+ 77us) server_negotiation.cc:407] Connection header received
0925 22:47:16.588118 (+ 869us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:47:16.588143 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:47:16.588541 (+ 398us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:47:16.588971 (+ 430us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:47:16.590093 (+ 1122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:47:16.592042 (+ 1949us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:47:16.593523 (+ 1481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:47:16.594331 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:47:16.594644 (+ 313us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:47:16.597789 (+ 3145us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:47:16.597799 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:47:16.597830 (+ 31us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:47:16.597889 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:47:16.600261 (+ 2372us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:47:16.600961 (+ 700us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:47:16.600966 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:47:16.600973 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:47:16.601143 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:47:16.601579 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:47:16.601583 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:47:16.601585 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:47:16.602221 (+ 636us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:47:16.602450 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0925 22:47:16.602855 (+ 405us) server_negotiation.cc:299] Negotiation successful
0925 22:47:16.603212 (+ 357us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":360,"thread_start_us":153,"threads_started":1}
I20260925 22:47:16.604449 24841 raft_consensus.cc:686] T 00000000000000000000000000000000 P 27eb88732a4e4caaae541c787b446b72 [term 1 LEADER]: Becoming Leader. State: Replica: 27eb88732a4e4caaae541c787b446b72, State: Running, Role: LEADER
I20260925 22:47:16.604833 24841 consensus_queue.cc:237] T 00000000000000000000000000000000 P 27eb88732a4e4caaae541c787b446b72 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "27eb88732a4e4caaae541c787b446b72" member_type: VOTER last_known_addr { host: "127.10.198.62" port: 42687 } }
I20260925 22:47:16.605221 24836 sys_catalog.cc:564] T 00000000000000000000000000000000 P 27eb88732a4e4caaae541c787b446b72 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 22:47:16.606254 24841 sys_catalog.cc:455] T 00000000000000000000000000000000 P 27eb88732a4e4caaae541c787b446b72 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 27eb88732a4e4caaae541c787b446b72. Latest consensus state: current_term: 1 leader_uuid: "27eb88732a4e4caaae541c787b446b72" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "27eb88732a4e4caaae541c787b446b72" member_type: VOTER last_known_addr { host: "127.10.198.62" port: 42687 } } }
I20260925 22:47:16.606410 24841 sys_catalog.cc:458] T 00000000000000000000000000000000 P 27eb88732a4e4caaae541c787b446b72 [sys.catalog]: This master's current role is: LEADER
I20260925 22:47:16.606956 24850 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 22:47:16.607604 24842 sys_catalog.cc:455] T 00000000000000000000000000000000 P 27eb88732a4e4caaae541c787b446b72 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "27eb88732a4e4caaae541c787b446b72" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "27eb88732a4e4caaae541c787b446b72" member_type: VOTER last_known_addr { host: "127.10.198.62" port: 42687 } } }
I20260925 22:47:16.607761 24842 sys_catalog.cc:458] T 00000000000000000000000000000000 P 27eb88732a4e4caaae541c787b446b72 [sys.catalog]: This master's current role is: LEADER
I20260925 22:47:16.610630 24850 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 22:47:16.617862 24850 catalog_manager.cc:1347] Generated new cluster ID: f4e6616644f449c08dd601a6389da1b3
I20260925 22:47:16.617965 24850 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 22:47:16.633381 24850 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 22:47:16.634891 24850 catalog_manager.cc:1504] Loading token signing keys...
I20260925 22:47:16.655534 24850 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 27eb88732a4e4caaae541c787b446b72: Generated new TSK 0
I20260925 22:47:16.656423 24850 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.10.198.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.198.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.198.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376259650642-11032-0/krb5kdc/kudu_127.10.198.1.keytab.
Entry for principal kudu/127.10.198.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376259650642-11032-0/krb5kdc/kudu_127.10.198.1.keytab.
I20260925 22:47:16.697140 11032 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrWXm8X/build/debug/bin/kudu
/tmp/dist-test-taskrWXm8X/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376259650642-11032-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376259650642-11032-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376259650642-11032-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376259650642-11032-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.198.1:0
--local_ip_for_outbound_sockets=127.10.198.1
--webserver_interface=127.10.198.1
--webserver_port=0
--tserver_master_addrs=127.10.198.62:42687
--keytab_file=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376259650642-11032-0/krb5kdc/kudu_127.10.198.1.keytab
--principal=kudu/127.10.198.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.198.20:44287
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376259650642-11032-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376259650642-11032-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376259650642-11032-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:47:16.821206 24862 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:47:16.821534 24862 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:47:16.821583 24862 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:47:16.826202 24862 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:47:16.826290 24862 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:47:16.826356 24862 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.198.1
I20260925 22:47:16.832271 24862 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.198.20:44287
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376259650642-11032-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376259650642-11032-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376259650642-11032-0/krb5kdc/kudu_127.10.198.1.keytab
--principal=kudu/127.10.198.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.198.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376259650642-11032-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.198.1
--webserver_port=0
--tserver_master_addrs=127.10.198.62:42687
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24862
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.198.1
--log_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376259650642-11032-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:09:16 UTC on ccycloud.quasar-afmhyw.root.comops.site
build id 4673379
I20260925 22:47:16.833534 24862 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:47:16.834729 24862 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:47:16.846019 24870 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:47:16.846212 24868 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:47:16.846259 24862 server_base.cc:1034] running on GCE node
W20260925 22:47:16.846503 24867 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:47:16.847118 24862 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:47:16.848080 24862 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:47:16.849293 24862 hybrid_clock.cc:648] HybridClock initialized: now 1790376436849253 us; error 68 us; skew 500 ppm
Sep 25 22:47:16 dist-test-slave-5vdr krb5kdc[24755](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376436, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.198.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:47:16.852916 24862 init.cc:377] Logged in from keytab as kudu/127.10.198.1@KRBTEST.COM (short username kudu)
I20260925 22:47:16.854287 24862 webserver.cc:460] Webserver started at http://127.10.198.1:35377/ using document root <none> and password file <none>
I20260925 22:47:16.854984 24862 fs_manager.cc:359] Metadata directory not provided
I20260925 22:47:16.855070 24862 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:47:16.855342 24862 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:47:16.857856 24862 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376259650642-11032-0/minicluster-data/ts-0/data/instance:
uuid: "d446d20efa5e403c9ca66ad9e6bd6829"
format_stamp: "Formatted at 2026-09-25 22:47:16 on dist-test-slave-5vdr"
server_key: "d2f2d829885eb12f22e8b0b03c4559fa"
server_key_iv: "39ca185db2e434a48e3305f4d90043b9"
server_key_version: "encryptionkey@0"
I20260925 22:47:16.858589 24862 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376259650642-11032-0/minicluster-data/ts-0/wal/instance:
uuid: "d446d20efa5e403c9ca66ad9e6bd6829"
format_stamp: "Formatted at 2026-09-25 22:47:16 on dist-test-slave-5vdr"
server_key: "d2f2d829885eb12f22e8b0b03c4559fa"
server_key_iv: "39ca185db2e434a48e3305f4d90043b9"
server_key_version: "encryptionkey@0"
I20260925 22:47:16.863399 24862 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.005s
I20260925 22:47:16.866791 24877 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:47:16.868453 24862 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 22:47:16.868634 24862 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376259650642-11032-0/minicluster-data/ts-0/data,/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376259650642-11032-0/minicluster-data/ts-0/wal
uuid: "d446d20efa5e403c9ca66ad9e6bd6829"
format_stamp: "Formatted at 2026-09-25 22:47:16 on dist-test-slave-5vdr"
server_key: "d2f2d829885eb12f22e8b0b03c4559fa"
server_key_iv: "39ca185db2e434a48e3305f4d90043b9"
server_key_version: "encryptionkey@0"
I20260925 22:47:16.868791 24862 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376259650642-11032-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376259650642-11032-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376259650642-11032-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:47:16.891070 24862 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:47:16.895287 24862 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:47:16.895536 24862 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:47:16.896322 24862 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 22:47:16.897277 24862 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:47:16.897330 24862 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:47:16.897399 24862 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:47:16.897433 24862 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 22:47:16.908115 24862 rpc_server.cc:307] RPC server started. Bound to: 127.10.198.1:35361
I20260925 22:47:16.908133 24990 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.198.1:35361 every 8 connection(s)
I20260925 22:47:16.909164 24862 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376259650642-11032-0/minicluster-data/ts-0/data/info.pb
I20260925 22:47:16.915297 11032 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrWXm8X/build/debug/bin/kudu as pid 24862
I20260925 22:47:16.915474 11032 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376259650642-11032-0/minicluster-data/ts-0/wal/instance
I20260925 22:47:16.915794 11032 external_mini_cluster.cc:1428] Setting key f8d8f203a2749b0508c29a9a166f73d0
Sep 25 22:47:16 dist-test-slave-5vdr krb5kdc[24755](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376436, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.198.1@KRBTEST.COM for kudu/127.10.198.62@KRBTEST.COM
I20260925 22:47:16.926745 24839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:47:16.911185 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.198.1:50479 (local address 127.10.198.62:42687)
0925 22:47:16.911352 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0925 22:47:16.911356 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:47:16.912438 (+ 1082us) server_negotiation.cc:407] Connection header received
0925 22:47:16.913886 (+ 1448us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:47:16.913888 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:47:16.913946 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:47:16.914036 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:47:16.915683 (+ 1647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:47:16.916412 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:47:16.918246 (+ 1834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:47:16.918496 (+ 250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:47:16.918622 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:47:16.922190 (+ 3568us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:47:16.922197 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:47:16.922203 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:47:16.922245 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:47:16.924190 (+ 1945us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:47:16.924870 (+ 680us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:47:16.924874 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:47:16.924876 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:47:16.924990 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:47:16.925386 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:47:16.925391 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:47:16.925395 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:47:16.925670 (+ 275us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:47:16.925801 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0925 22:47:16.926362 (+ 561us) server_negotiation.cc:299] Negotiation successful
0925 22:47:16.926529 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
WARNING: no policy specified for kudu/127.10.198.2@KRBTEST.COM; defaulting to no policy
I20260925 22:47:16.927577 24993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:47:16.911698 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.198.62:42687 (local address 127.10.198.1:50479)
0925 22:47:16.912246 (+ 548us) negotiation.cc:107] Waiting for socket to connect
0925 22:47:16.912299 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0925 22:47:16.913616 (+ 1317us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:47:16.914226 (+ 610us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:47:16.914234 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:47:16.914649 (+ 415us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:47:16.915391 (+ 742us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:47:16.915421 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:47:16.916585 (+ 1164us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:47:16.916589 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:47:16.918066 (+ 1477us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:47:16.918086 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:47:16.918701 (+ 615us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:47:16.918708 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:47:16.918989 (+ 281us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:47:16.919933 (+ 944us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:47:16.919961 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:47:16.921920 (+ 1959us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:47:16.924386 (+ 2466us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:47:16.924393 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:47:16.924410 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:47:16.924736 (+ 326us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:47:16.925099 (+ 363us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:47:16.925102 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:47:16.925103 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:47:16.925264 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:47:16.925796 (+ 532us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:47:16.925803 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:47:16.926080 (+ 277us) client_negotiation.cc:769] Sending connection context
0925 22:47:16.926258 (+ 178us) client_negotiation.cc:240] Negotiation successful
0925 22:47:16.926571 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":330,"thread_start_us":169,"threads_started":1}
I20260925 22:47:16.929225 24991 heartbeater.cc:347] Connected to a master server at 127.10.198.62:42687
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.198.2@KRBTEST.COM" created.
I20260925 22:47:16.929631 24991 heartbeater.cc:464] Registering TS with master...
I20260925 22:47:16.930356 24991 heartbeater.cc:511] Master 127.10.198.62:42687 requested a full tablet report, sending...
I20260925 22:47:16.932499 24797 ts_manager.cc:194] Registered new tserver with Master: d446d20efa5e403c9ca66ad9e6bd6829 (127.10.198.1:35361)
I20260925 22:47:16.934383 24797 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.198.1@KRBTEST.COM'} at 127.10.198.1:50479
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.198.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376259650642-11032-0/krb5kdc/kudu_127.10.198.2.keytab.
Entry for principal kudu/127.10.198.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376259650642-11032-0/krb5kdc/kudu_127.10.198.2.keytab.
I20260925 22:47:16.947551 11032 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrWXm8X/build/debug/bin/kudu
/tmp/dist-test-taskrWXm8X/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376259650642-11032-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376259650642-11032-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376259650642-11032-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376259650642-11032-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.198.2:0
--local_ip_for_outbound_sockets=127.10.198.2
--webserver_interface=127.10.198.2
--webserver_port=0
--tserver_master_addrs=127.10.198.62:42687
--keytab_file=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376259650642-11032-0/krb5kdc/kudu_127.10.198.2.keytab
--principal=kudu/127.10.198.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.198.20:44287
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376259650642-11032-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376259650642-11032-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376259650642-11032-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:47:17.067138 24996 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:47:17.067472 24996 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:47:17.067520 24996 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:47:17.071975 24996 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:47:17.072058 24996 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:47:17.072168 24996 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.198.2
I20260925 22:47:17.078231 24996 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.198.20:44287
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376259650642-11032-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376259650642-11032-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376259650642-11032-0/krb5kdc/kudu_127.10.198.2.keytab
--principal=kudu/127.10.198.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.198.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376259650642-11032-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.198.2
--webserver_port=0
--tserver_master_addrs=127.10.198.62:42687
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24996
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.198.2
--log_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376259650642-11032-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:09:16 UTC on ccycloud.quasar-afmhyw.root.comops.site
build id 4673379
I20260925 22:47:17.079478 24996 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:47:17.080773 24996 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:47:17.091770 25004 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:47:17.091770 25002 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:47:17.092052 24996 server_base.cc:1034] running on GCE node
W20260925 22:47:17.091940 25001 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:47:17.092626 24996 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:47:17.093403 24996 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:47:17.094635 24996 hybrid_clock.cc:648] HybridClock initialized: now 1790376437094576 us; error 78 us; skew 500 ppm
Sep 25 22:47:17 dist-test-slave-5vdr krb5kdc[24755](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376437, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.198.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:47:17.097993 24996 init.cc:377] Logged in from keytab as kudu/127.10.198.2@KRBTEST.COM (short username kudu)
I20260925 22:47:17.099116 24996 webserver.cc:460] Webserver started at http://127.10.198.2:46157/ using document root <none> and password file <none>
I20260925 22:47:17.099750 24996 fs_manager.cc:359] Metadata directory not provided
I20260925 22:47:17.099830 24996 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:47:17.100128 24996 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:47:17.102679 24996 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376259650642-11032-0/minicluster-data/ts-1/data/instance:
uuid: "e2485ed683004618a230d0a403a4d008"
format_stamp: "Formatted at 2026-09-25 22:47:17 on dist-test-slave-5vdr"
server_key: "796a5bd9100cbda6ca63715963ec177c"
server_key_iv: "710b02d7903df43cd507750f807f85d8"
server_key_version: "encryptionkey@0"
I20260925 22:47:17.103499 24996 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376259650642-11032-0/minicluster-data/ts-1/wal/instance:
uuid: "e2485ed683004618a230d0a403a4d008"
format_stamp: "Formatted at 2026-09-25 22:47:17 on dist-test-slave-5vdr"
server_key: "796a5bd9100cbda6ca63715963ec177c"
server_key_iv: "710b02d7903df43cd507750f807f85d8"
server_key_version: "encryptionkey@0"
I20260925 22:47:17.108423 24996 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260925 22:47:17.111307 25011 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:47:17.112880 24996 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 22:47:17.113072 24996 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376259650642-11032-0/minicluster-data/ts-1/data,/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376259650642-11032-0/minicluster-data/ts-1/wal
uuid: "e2485ed683004618a230d0a403a4d008"
format_stamp: "Formatted at 2026-09-25 22:47:17 on dist-test-slave-5vdr"
server_key: "796a5bd9100cbda6ca63715963ec177c"
server_key_iv: "710b02d7903df43cd507750f807f85d8"
server_key_version: "encryptionkey@0"
I20260925 22:47:17.113219 24996 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376259650642-11032-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376259650642-11032-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376259650642-11032-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:47:17.129243 24996 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:47:17.133191 24996 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:47:17.133543 24996 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:47:17.134234 24996 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 22:47:17.135203 24996 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:47:17.135282 24996 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:47:17.135351 24996 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:47:17.135401 24996 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 22:47:17.145649 24996 rpc_server.cc:307] RPC server started. Bound to: 127.10.198.2:35091
I20260925 22:47:17.145671 25124 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.198.2:35091 every 8 connection(s)
I20260925 22:47:17.146839 24996 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376259650642-11032-0/minicluster-data/ts-1/data/info.pb
I20260925 22:47:17.155395 11032 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrWXm8X/build/debug/bin/kudu as pid 24996
I20260925 22:47:17.155589 11032 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376259650642-11032-0/minicluster-data/ts-1/wal/instance
I20260925 22:47:17.156009 11032 external_mini_cluster.cc:1428] Setting key 534071f33a26978ce0495b7349c63d56
Sep 25 22:47:17 dist-test-slave-5vdr krb5kdc[24755](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376437, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.198.2@KRBTEST.COM for kudu/127.10.198.62@KRBTEST.COM
I20260925 22:47:17.166167 24839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:47:17.149288 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.198.2:47613 (local address 127.10.198.62:42687)
0925 22:47:17.149488 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0925 22:47:17.149492 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:47:17.150579 (+ 1087us) server_negotiation.cc:407] Connection header received
0925 22:47:17.152323 (+ 1744us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:47:17.152326 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:47:17.152388 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:47:17.152480 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:47:17.154089 (+ 1609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:47:17.154667 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:47:17.156301 (+ 1634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:47:17.156671 (+ 370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:47:17.156855 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:47:17.160460 (+ 3605us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:47:17.160465 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:47:17.160470 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:47:17.160514 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:47:17.162494 (+ 1980us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:47:17.163544 (+ 1050us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:47:17.163549 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:47:17.163550 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:47:17.163684 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:47:17.164317 (+ 633us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:47:17.164320 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:47:17.164321 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:47:17.164577 (+ 256us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:47:17.164699 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0925 22:47:17.165815 (+ 1116us) server_negotiation.cc:299] Negotiation successful
0925 22:47:17.166002 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93}
I20260925 22:47:17.167457 25127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:47:17.149726 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.198.62:42687 (local address 127.10.198.2:47613)
0925 22:47:17.150353 (+ 627us) negotiation.cc:107] Waiting for socket to connect
0925 22:47:17.150416 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0925 22:47:17.151997 (+ 1581us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:47:17.152738 (+ 741us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:47:17.152747 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:47:17.153227 (+ 480us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:47:17.153868 (+ 641us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:47:17.153887 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:47:17.154864 (+ 977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:47:17.154867 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:47:17.156070 (+ 1203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:47:17.156085 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:47:17.156980 (+ 895us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:47:17.156987 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:47:17.157248 (+ 261us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:47:17.158163 (+ 915us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:47:17.158193 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:47:17.160174 (+ 1981us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:47:17.162723 (+ 2549us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:47:17.162736 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:47:17.162762 (+ 26us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:47:17.163293 (+ 531us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:47:17.163832 (+ 539us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:47:17.163838 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:47:17.163864 (+ 26us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:47:17.164155 (+ 291us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:47:17.164723 (+ 568us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:47:17.164735 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:47:17.165323 (+ 588us) client_negotiation.cc:769] Sending connection context
0925 22:47:17.165620 (+ 297us) client_negotiation.cc:240] Negotiation successful
0925 22:47:17.166095 (+ 475us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":369,"thread_start_us":163,"threads_started":1}
WARNING: no policy specified for kudu/127.10.198.3@KRBTEST.COM; defaulting to no policy
I20260925 22:47:17.169140 25125 heartbeater.cc:347] Connected to a master server at 127.10.198.62:42687
I20260925 22:47:17.169564 25125 heartbeater.cc:464] Registering TS with master...
I20260925 22:47:17.170222 25125 heartbeater.cc:511] Master 127.10.198.62:42687 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.198.3@KRBTEST.COM" created.
I20260925 22:47:17.171815 24797 ts_manager.cc:194] Registered new tserver with Master: e2485ed683004618a230d0a403a4d008 (127.10.198.2:35091)
I20260925 22:47:17.172592 24797 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.198.2@KRBTEST.COM'} at 127.10.198.2:47613
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.198.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376259650642-11032-0/krb5kdc/kudu_127.10.198.3.keytab.
Entry for principal kudu/127.10.198.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376259650642-11032-0/krb5kdc/kudu_127.10.198.3.keytab.
I20260925 22:47:17.190621 11032 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrWXm8X/build/debug/bin/kudu
/tmp/dist-test-taskrWXm8X/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376259650642-11032-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376259650642-11032-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376259650642-11032-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376259650642-11032-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.198.3:0
--local_ip_for_outbound_sockets=127.10.198.3
--webserver_interface=127.10.198.3
--webserver_port=0
--tserver_master_addrs=127.10.198.62:42687
--keytab_file=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376259650642-11032-0/krb5kdc/kudu_127.10.198.3.keytab
--principal=kudu/127.10.198.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.198.20:44287
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376259650642-11032-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376259650642-11032-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376259650642-11032-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:47:17.314136 25130 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:47:17.314471 25130 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:47:17.314517 25130 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:47:17.319057 25130 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:47:17.319152 25130 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:47:17.319218 25130 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.198.3
I20260925 22:47:17.325181 25130 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.198.20:44287
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376259650642-11032-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376259650642-11032-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376259650642-11032-0/krb5kdc/kudu_127.10.198.3.keytab
--principal=kudu/127.10.198.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.198.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376259650642-11032-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.198.3
--webserver_port=0
--tserver_master_addrs=127.10.198.62:42687
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25130
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.198.3
--log_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376259650642-11032-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:09:16 UTC on ccycloud.quasar-afmhyw.root.comops.site
build id 4673379
I20260925 22:47:17.326340 25130 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:47:17.327386 25130 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:47:17.337693 25136 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:47:17.337759 25138 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:47:17.337769 25135 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:47:17.338176 25130 server_base.cc:1034] running on GCE node
I20260925 22:47:17.338755 25130 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:47:17.339532 25130 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:47:17.340773 25130 hybrid_clock.cc:648] HybridClock initialized: now 1790376437340744 us; error 45 us; skew 500 ppm
Sep 25 22:47:17 dist-test-slave-5vdr krb5kdc[24755](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376437, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.198.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:47:17.344303 25130 init.cc:377] Logged in from keytab as kudu/127.10.198.3@KRBTEST.COM (short username kudu)
I20260925 22:47:17.345500 25130 webserver.cc:460] Webserver started at http://127.10.198.3:42855/ using document root <none> and password file <none>
I20260925 22:47:17.346225 25130 fs_manager.cc:359] Metadata directory not provided
I20260925 22:47:17.346309 25130 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:47:17.346580 25130 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:47:17.349249 25130 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376259650642-11032-0/minicluster-data/ts-2/data/instance:
uuid: "e6fad9ce2c614cb08498cd3cdf453424"
format_stamp: "Formatted at 2026-09-25 22:47:17 on dist-test-slave-5vdr"
server_key: "1ebdbebba049bc1ce205275c6fad909d"
server_key_iv: "f7e50512d36c1fc08f3d67e35c186e1f"
server_key_version: "encryptionkey@0"
I20260925 22:47:17.349983 25130 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376259650642-11032-0/minicluster-data/ts-2/wal/instance:
uuid: "e6fad9ce2c614cb08498cd3cdf453424"
format_stamp: "Formatted at 2026-09-25 22:47:17 on dist-test-slave-5vdr"
server_key: "1ebdbebba049bc1ce205275c6fad909d"
server_key_iv: "f7e50512d36c1fc08f3d67e35c186e1f"
server_key_version: "encryptionkey@0"
I20260925 22:47:17.354737 25130 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260925 22:47:17.358023 25145 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:47:17.359717 25130 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260925 22:47:17.359935 25130 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376259650642-11032-0/minicluster-data/ts-2/data,/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376259650642-11032-0/minicluster-data/ts-2/wal
uuid: "e6fad9ce2c614cb08498cd3cdf453424"
format_stamp: "Formatted at 2026-09-25 22:47:17 on dist-test-slave-5vdr"
server_key: "1ebdbebba049bc1ce205275c6fad909d"
server_key_iv: "f7e50512d36c1fc08f3d67e35c186e1f"
server_key_version: "encryptionkey@0"
I20260925 22:47:17.360065 25130 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376259650642-11032-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376259650642-11032-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376259650642-11032-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:47:17.392655 25130 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:47:17.396291 25130 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:47:17.396505 25130 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:47:17.397105 25130 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 22:47:17.398098 25130 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:47:17.398170 25130 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:47:17.398241 25130 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:47:17.398259 25130 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 22:47:17.408599 25130 rpc_server.cc:307] RPC server started. Bound to: 127.10.198.3:37747
I20260925 22:47:17.408699 25258 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.198.3:37747 every 8 connection(s)
I20260925 22:47:17.409795 25130 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376259650642-11032-0/minicluster-data/ts-2/data/info.pb
I20260925 22:47:17.418968 11032 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrWXm8X/build/debug/bin/kudu as pid 25130
I20260925 22:47:17.419124 11032 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376259650642-11032-0/minicluster-data/ts-2/wal/instance
I20260925 22:47:17.419476 11032 external_mini_cluster.cc:1428] Setting key 349794918a639636c82f0d764587bab7
Sep 25 22:47:17 dist-test-slave-5vdr krb5kdc[24755](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376437, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.198.3@KRBTEST.COM for kudu/127.10.198.62@KRBTEST.COM
I20260925 22:47:17.426389 24839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:47:17.412215 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.198.3:41967 (local address 127.10.198.62:42687)
0925 22:47:17.412401 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0925 22:47:17.412405 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:47:17.413540 (+ 1135us) server_negotiation.cc:407] Connection header received
0925 22:47:17.415065 (+ 1525us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:47:17.415068 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:47:17.415135 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:47:17.415220 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:47:17.416690 (+ 1470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:47:17.417354 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:47:17.418828 (+ 1474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:47:17.419077 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:47:17.419215 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:47:17.422656 (+ 3441us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:47:17.422659 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:47:17.422661 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:47:17.422690 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:47:17.424107 (+ 1417us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:47:17.424743 (+ 636us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:47:17.424745 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:47:17.424746 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:47:17.424826 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:47:17.425171 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:47:17.425173 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:47:17.425174 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:47:17.425361 (+ 187us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:47:17.425441 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0925 22:47:17.426115 (+ 674us) server_negotiation.cc:299] Negotiation successful
0925 22:47:17.426253 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260925 22:47:17.427351 25261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:47:17.412734 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.198.62:42687 (local address 127.10.198.3:41967)
0925 22:47:17.413359 (+ 625us) negotiation.cc:107] Waiting for socket to connect
0925 22:47:17.413413 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0925 22:47:17.414791 (+ 1378us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:47:17.415426 (+ 635us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:47:17.415434 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:47:17.415939 (+ 505us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:47:17.416504 (+ 565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:47:17.416521 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:47:17.417496 (+ 975us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:47:17.417499 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:47:17.418679 (+ 1180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:47:17.418691 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:47:17.419343 (+ 652us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:47:17.419349 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:47:17.419597 (+ 248us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:47:17.420379 (+ 782us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:47:17.420402 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:47:17.422477 (+ 2075us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:47:17.424260 (+ 1783us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:47:17.424267 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:47:17.424284 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:47:17.424644 (+ 360us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:47:17.424928 (+ 284us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:47:17.424930 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:47:17.424931 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:47:17.425087 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:47:17.425488 (+ 401us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:47:17.425497 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:47:17.425816 (+ 319us) client_negotiation.cc:769] Sending connection context
0925 22:47:17.426015 (+ 199us) client_negotiation.cc:240] Negotiation successful
0925 22:47:17.426324 (+ 309us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":354,"thread_start_us":143,"threads_started":1}
I20260925 22:47:17.428803 25259 heartbeater.cc:347] Connected to a master server at 127.10.198.62:42687
I20260925 22:47:17.429092 25259 heartbeater.cc:464] Registering TS with master...
I20260925 22:47:17.429677 25259 heartbeater.cc:511] Master 127.10.198.62:42687 requested a full tablet report, sending...
I20260925 22:47:17.430951 24797 ts_manager.cc:194] Registered new tserver with Master: e6fad9ce2c614cb08498cd3cdf453424 (127.10.198.3:37747)
I20260925 22:47:17.431588 24797 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.198.3@KRBTEST.COM'} at 127.10.198.3:41967
I20260925 22:47:17.434360 11032 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 22:47:17.444720 24839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:47:17.436493 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38602 (local address 127.10.198.62:42687)
0925 22:47:17.436640 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0925 22:47:17.436645 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:47:17.436831 (+ 186us) server_negotiation.cc:407] Connection header received
0925 22:47:17.436997 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:47:17.437000 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:47:17.437057 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:47:17.437148 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:47:17.437910 (+ 762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:47:17.438476 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:47:17.439366 (+ 890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:47:17.439606 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:47:17.439723 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:47:17.441222 (+ 1499us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:47:17.441226 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:47:17.441229 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:47:17.441272 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:47:17.442763 (+ 1491us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:47:17.443275 (+ 512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:47:17.443279 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:47:17.443281 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:47:17.443376 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:47:17.443668 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:47:17.443672 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:47:17.443674 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:47:17.444010 (+ 336us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:47:17.444172 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0925 22:47:17.444422 (+ 250us) server_negotiation.cc:299] Negotiation successful
0925 22:47:17.444547 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260925 22:47:17.448163 24797 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38602:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 22:47:17.451400 24797 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 22:47:17.468667 25273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:47:17.461855 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42292 (local address 127.10.198.3:37747)
0925 22:47:17.462337 (+ 482us) server_negotiation.cc:206] Beginning negotiation
0925 22:47:17.462349 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 22:47:17.462781 (+ 432us) server_negotiation.cc:407] Connection header received
0925 22:47:17.462999 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:47:17.463006 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:47:17.463260 (+ 254us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:47:17.463431 (+ 171us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:47:17.464075 (+ 644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:47:17.465245 (+ 1170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:47:17.467038 (+ 1793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:47:17.467789 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:47:17.467980 (+ 191us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:47:17.468169 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0925 22:47:17.468373 (+ 204us) server_negotiation.cc:299] Negotiation successful
0925 22:47:17.468485 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":372,"thread_start_us":155,"threads_started":1}
I20260925 22:47:17.468652 25270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:47:17.461691 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.198.3:37747 (local address 127.0.0.1:42292)
0925 22:47:17.462610 (+ 919us) negotiation.cc:107] Waiting for socket to connect
0925 22:47:17.462676 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0925 22:47:17.462827 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:47:17.463624 (+ 797us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:47:17.463630 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:47:17.463650 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:47:17.463904 (+ 254us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:47:17.463911 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:47:17.465656 (+ 1745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:47:17.465660 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:47:17.466878 (+ 1218us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:47:17.466888 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:47:17.467996 (+ 1108us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:47:17.467999 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:47:17.468131 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:47:17.468172 (+ 41us) client_negotiation.cc:769] Sending connection context
0925 22:47:17.468331 (+ 159us) client_negotiation.cc:240] Negotiation successful
0925 22:47:17.468489 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":735,"mutex_wait_us":52,"thread_start_us":112,"threads_started":1}
I20260925 22:47:17.469240 25274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:47:17.462177 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.198.1:35361 (local address 127.0.0.1:39374)
0925 22:47:17.462928 (+ 751us) negotiation.cc:107] Waiting for socket to connect
0925 22:47:17.462942 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 22:47:17.463034 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:47:17.463467 (+ 433us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:47:17.463473 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:47:17.463498 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:47:17.463744 (+ 246us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:47:17.463754 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:47:17.466028 (+ 2274us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:47:17.466032 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:47:17.467207 (+ 1175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:47:17.467219 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:47:17.468599 (+ 1380us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:47:17.468603 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:47:17.468703 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:47:17.468887 (+ 184us) client_negotiation.cc:769] Sending connection context
0925 22:47:17.468996 (+ 109us) client_negotiation.cc:240] Negotiation successful
0925 22:47:17.469086 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":659,"mutex_wait_us":40,"thread_start_us":92,"threads_started":1}
I20260925 22:47:17.469308 25275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:47:17.462325 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39374 (local address 127.10.198.1:35361)
0925 22:47:17.462629 (+ 304us) server_negotiation.cc:206] Beginning negotiation
0925 22:47:17.462635 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 22:47:17.463016 (+ 381us) server_negotiation.cc:407] Connection header received
0925 22:47:17.463153 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:47:17.463157 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:47:17.463304 (+ 147us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:47:17.463431 (+ 127us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:47:17.463935 (+ 504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:47:17.465533 (+ 1598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:47:17.467363 (+ 1830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:47:17.468279 (+ 916us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:47:17.468389 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:47:17.468529 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0925 22:47:17.469044 (+ 515us) server_negotiation.cc:299] Negotiation successful
0925 22:47:17.469160 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":211,"thread_start_us":141,"threads_started":1}
I20260925 22:47:17.470160 25272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:47:17.461290 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.198.2:35091 (local address 127.0.0.1:36942)
0925 22:47:17.462664 (+ 1374us) negotiation.cc:107] Waiting for socket to connect
0925 22:47:17.462678 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 22:47:17.462825 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:47:17.463680 (+ 855us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:47:17.463683 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:47:17.463700 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:47:17.463904 (+ 204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:47:17.463911 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:47:17.466842 (+ 2931us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:47:17.466846 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:47:17.467949 (+ 1103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:47:17.467961 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:47:17.469760 (+ 1799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:47:17.469763 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:47:17.469853 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:47:17.469874 (+ 21us) client_negotiation.cc:769] Sending connection context
0925 22:47:17.469935 (+ 61us) client_negotiation.cc:240] Negotiation successful
0925 22:47:17.470025 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":797,"mutex_wait_us":61,"thread_start_us":92,"threads_started":1}
I20260925 22:47:17.470213 25271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:47:17.461487 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36942 (local address 127.10.198.2:35091)
0925 22:47:17.463174 (+ 1687us) server_negotiation.cc:206] Beginning negotiation
0925 22:47:17.463183 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 22:47:17.463202 (+ 19us) server_negotiation.cc:407] Connection header received
0925 22:47:17.463265 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:47:17.463268 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:47:17.463554 (+ 286us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:47:17.463717 (+ 163us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:47:17.465476 (+ 1759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:47:17.466672 (+ 1196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:47:17.468111 (+ 1439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:47:17.469644 (+ 1533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:47:17.469765 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:47:17.469916 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0925 22:47:17.469982 (+ 66us) server_negotiation.cc:299] Negotiation successful
0925 22:47:17.470075 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1573,"thread_start_us":151,"threads_started":1}
I20260925 22:47:17.472936 25193 tablet_service.cc:1467] Processing CreateTablet for tablet d6ec8f4e57b94888b44230fa02d0c216 (DEFAULT_TABLE table=test-table [id=f1f12203d292477cac29d7fff1cedb52]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 22:47:17.472936 24925 tablet_service.cc:1467] Processing CreateTablet for tablet d6ec8f4e57b94888b44230fa02d0c216 (DEFAULT_TABLE table=test-table [id=f1f12203d292477cac29d7fff1cedb52]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 22:47:17.474287 24925 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d6ec8f4e57b94888b44230fa02d0c216. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:47:17.474288 25193 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d6ec8f4e57b94888b44230fa02d0c216. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:47:17.476663 25059 tablet_service.cc:1467] Processing CreateTablet for tablet d6ec8f4e57b94888b44230fa02d0c216 (DEFAULT_TABLE table=test-table [id=f1f12203d292477cac29d7fff1cedb52]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 22:47:17.477764 25059 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d6ec8f4e57b94888b44230fa02d0c216. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:47:17.482832 25276 tablet_bootstrap.cc:492] T d6ec8f4e57b94888b44230fa02d0c216 P d446d20efa5e403c9ca66ad9e6bd6829: Bootstrap starting.
I20260925 22:47:17.484285 25277 tablet_bootstrap.cc:492] T d6ec8f4e57b94888b44230fa02d0c216 P e6fad9ce2c614cb08498cd3cdf453424: Bootstrap starting.
I20260925 22:47:17.485342 25278 tablet_bootstrap.cc:492] T d6ec8f4e57b94888b44230fa02d0c216 P e2485ed683004618a230d0a403a4d008: Bootstrap starting.
I20260925 22:47:17.487723 25276 tablet_bootstrap.cc:654] T d6ec8f4e57b94888b44230fa02d0c216 P d446d20efa5e403c9ca66ad9e6bd6829: Neither blocks nor log segments found. Creating new log.
I20260925 22:47:17.488276 25278 tablet_bootstrap.cc:654] T d6ec8f4e57b94888b44230fa02d0c216 P e2485ed683004618a230d0a403a4d008: Neither blocks nor log segments found. Creating new log.
I20260925 22:47:17.488874 25276 log.cc:824] T d6ec8f4e57b94888b44230fa02d0c216 P d446d20efa5e403c9ca66ad9e6bd6829: Log is configured to *not* fsync() on all Append() calls
I20260925 22:47:17.489065 25278 log.cc:824] T d6ec8f4e57b94888b44230fa02d0c216 P e2485ed683004618a230d0a403a4d008: Log is configured to *not* fsync() on all Append() calls
I20260925 22:47:17.490167 25277 tablet_bootstrap.cc:654] T d6ec8f4e57b94888b44230fa02d0c216 P e6fad9ce2c614cb08498cd3cdf453424: Neither blocks nor log segments found. Creating new log.
I20260925 22:47:17.491274 25277 log.cc:824] T d6ec8f4e57b94888b44230fa02d0c216 P e6fad9ce2c614cb08498cd3cdf453424: Log is configured to *not* fsync() on all Append() calls
I20260925 22:47:17.491714 25276 tablet_bootstrap.cc:492] T d6ec8f4e57b94888b44230fa02d0c216 P d446d20efa5e403c9ca66ad9e6bd6829: No bootstrap required, opened a new log
I20260925 22:47:17.491833 25278 tablet_bootstrap.cc:492] T d6ec8f4e57b94888b44230fa02d0c216 P e2485ed683004618a230d0a403a4d008: No bootstrap required, opened a new log
I20260925 22:47:17.491994 25276 ts_tablet_manager.cc:1397] T d6ec8f4e57b94888b44230fa02d0c216 P d446d20efa5e403c9ca66ad9e6bd6829: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.004s
I20260925 22:47:17.492079 25278 ts_tablet_manager.cc:1397] T d6ec8f4e57b94888b44230fa02d0c216 P e2485ed683004618a230d0a403a4d008: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.005s
I20260925 22:47:17.493673 25277 tablet_bootstrap.cc:492] T d6ec8f4e57b94888b44230fa02d0c216 P e6fad9ce2c614cb08498cd3cdf453424: No bootstrap required, opened a new log
I20260925 22:47:17.493916 25277 ts_tablet_manager.cc:1397] T d6ec8f4e57b94888b44230fa02d0c216 P e6fad9ce2c614cb08498cd3cdf453424: Time spent bootstrapping tablet: real 0.011s user 0.005s sys 0.000s
I20260925 22:47:17.496845 25276 raft_consensus.cc:348] T d6ec8f4e57b94888b44230fa02d0c216 P d446d20efa5e403c9ca66ad9e6bd6829 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2485ed683004618a230d0a403a4d008" member_type: VOTER last_known_addr { host: "127.10.198.2" port: 35091 } } peers { permanent_uuid: "d446d20efa5e403c9ca66ad9e6bd6829" member_type: VOTER last_known_addr { host: "127.10.198.1" port: 35361 } } peers { permanent_uuid: "e6fad9ce2c614cb08498cd3cdf453424" member_type: VOTER last_known_addr { host: "127.10.198.3" port: 37747 } }
I20260925 22:47:17.496852 25278 raft_consensus.cc:348] T d6ec8f4e57b94888b44230fa02d0c216 P e2485ed683004618a230d0a403a4d008 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2485ed683004618a230d0a403a4d008" member_type: VOTER last_known_addr { host: "127.10.198.2" port: 35091 } } peers { permanent_uuid: "d446d20efa5e403c9ca66ad9e6bd6829" member_type: VOTER last_known_addr { host: "127.10.198.1" port: 35361 } } peers { permanent_uuid: "e6fad9ce2c614cb08498cd3cdf453424" member_type: VOTER last_known_addr { host: "127.10.198.3" port: 37747 } }
I20260925 22:47:17.497159 25276 raft_consensus.cc:374] T d6ec8f4e57b94888b44230fa02d0c216 P d446d20efa5e403c9ca66ad9e6bd6829 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:47:17.497187 25278 raft_consensus.cc:374] T d6ec8f4e57b94888b44230fa02d0c216 P e2485ed683004618a230d0a403a4d008 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:47:17.497223 25276 raft_consensus.cc:729] T d6ec8f4e57b94888b44230fa02d0c216 P d446d20efa5e403c9ca66ad9e6bd6829 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d446d20efa5e403c9ca66ad9e6bd6829, State: Initialized, Role: FOLLOWER
I20260925 22:47:17.497265 25278 raft_consensus.cc:729] T d6ec8f4e57b94888b44230fa02d0c216 P e2485ed683004618a230d0a403a4d008 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e2485ed683004618a230d0a403a4d008, State: Initialized, Role: FOLLOWER
I20260925 22:47:17.497746 25276 consensus_queue.cc:260] T d6ec8f4e57b94888b44230fa02d0c216 P d446d20efa5e403c9ca66ad9e6bd6829 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2485ed683004618a230d0a403a4d008" member_type: VOTER last_known_addr { host: "127.10.198.2" port: 35091 } } peers { permanent_uuid: "d446d20efa5e403c9ca66ad9e6bd6829" member_type: VOTER last_known_addr { host: "127.10.198.1" port: 35361 } } peers { permanent_uuid: "e6fad9ce2c614cb08498cd3cdf453424" member_type: VOTER last_known_addr { host: "127.10.198.3" port: 37747 } }
I20260925 22:47:17.497831 25278 consensus_queue.cc:260] T d6ec8f4e57b94888b44230fa02d0c216 P e2485ed683004618a230d0a403a4d008 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2485ed683004618a230d0a403a4d008" member_type: VOTER last_known_addr { host: "127.10.198.2" port: 35091 } } peers { permanent_uuid: "d446d20efa5e403c9ca66ad9e6bd6829" member_type: VOTER last_known_addr { host: "127.10.198.1" port: 35361 } } peers { permanent_uuid: "e6fad9ce2c614cb08498cd3cdf453424" member_type: VOTER last_known_addr { host: "127.10.198.3" port: 37747 } }
I20260925 22:47:17.498970 25276 ts_tablet_manager.cc:1428] T d6ec8f4e57b94888b44230fa02d0c216 P d446d20efa5e403c9ca66ad9e6bd6829: Time spent starting tablet: real 0.007s user 0.005s sys 0.000s
I20260925 22:47:17.499027 25278 ts_tablet_manager.cc:1428] T d6ec8f4e57b94888b44230fa02d0c216 P e2485ed683004618a230d0a403a4d008: Time spent starting tablet: real 0.007s user 0.000s sys 0.006s
I20260925 22:47:17.498771 25277 raft_consensus.cc:348] T d6ec8f4e57b94888b44230fa02d0c216 P e6fad9ce2c614cb08498cd3cdf453424 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2485ed683004618a230d0a403a4d008" member_type: VOTER last_known_addr { host: "127.10.198.2" port: 35091 } } peers { permanent_uuid: "d446d20efa5e403c9ca66ad9e6bd6829" member_type: VOTER last_known_addr { host: "127.10.198.1" port: 35361 } } peers { permanent_uuid: "e6fad9ce2c614cb08498cd3cdf453424" member_type: VOTER last_known_addr { host: "127.10.198.3" port: 37747 } }
I20260925 22:47:17.499130 25277 raft_consensus.cc:374] T d6ec8f4e57b94888b44230fa02d0c216 P e6fad9ce2c614cb08498cd3cdf453424 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:47:17.499204 25277 raft_consensus.cc:729] T d6ec8f4e57b94888b44230fa02d0c216 P e6fad9ce2c614cb08498cd3cdf453424 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e6fad9ce2c614cb08498cd3cdf453424, State: Initialized, Role: FOLLOWER
I20260925 22:47:17.499377 24991 heartbeater.cc:503] Master 127.10.198.62:42687 was elected leader, sending a full tablet report...
I20260925 22:47:17.499435 25125 heartbeater.cc:503] Master 127.10.198.62:42687 was elected leader, sending a full tablet report...
I20260925 22:47:17.499718 25277 consensus_queue.cc:260] T d6ec8f4e57b94888b44230fa02d0c216 P e6fad9ce2c614cb08498cd3cdf453424 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2485ed683004618a230d0a403a4d008" member_type: VOTER last_known_addr { host: "127.10.198.2" port: 35091 } } peers { permanent_uuid: "d446d20efa5e403c9ca66ad9e6bd6829" member_type: VOTER last_known_addr { host: "127.10.198.1" port: 35361 } } peers { permanent_uuid: "e6fad9ce2c614cb08498cd3cdf453424" member_type: VOTER last_known_addr { host: "127.10.198.3" port: 37747 } }
I20260925 22:47:17.500543 25259 heartbeater.cc:503] Master 127.10.198.62:42687 was elected leader, sending a full tablet report...
I20260925 22:47:17.501011 25277 ts_tablet_manager.cc:1428] T d6ec8f4e57b94888b44230fa02d0c216 P e6fad9ce2c614cb08498cd3cdf453424: Time spent starting tablet: real 0.007s user 0.008s sys 0.000s
I20260925 22:47:17.582432 25282 raft_consensus.cc:482] T d6ec8f4e57b94888b44230fa02d0c216 P d446d20efa5e403c9ca66ad9e6bd6829 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 22:47:17.582716 25282 raft_consensus.cc:504] T d6ec8f4e57b94888b44230fa02d0c216 P d446d20efa5e403c9ca66ad9e6bd6829 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2485ed683004618a230d0a403a4d008" member_type: VOTER last_known_addr { host: "127.10.198.2" port: 35091 } } peers { permanent_uuid: "d446d20efa5e403c9ca66ad9e6bd6829" member_type: VOTER last_known_addr { host: "127.10.198.1" port: 35361 } } peers { permanent_uuid: "e6fad9ce2c614cb08498cd3cdf453424" member_type: VOTER last_known_addr { host: "127.10.198.3" port: 37747 } }
I20260925 22:47:17.584146 25282 leader_election.cc:288] T d6ec8f4e57b94888b44230fa02d0c216 P d446d20efa5e403c9ca66ad9e6bd6829 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e2485ed683004618a230d0a403a4d008 (127.10.198.2:35091), e6fad9ce2c614cb08498cd3cdf453424 (127.10.198.3:37747)
I20260925 22:47:17.589962 25273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:47:17.584920 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.198.1:55511 (local address 127.10.198.3:37747)
0925 22:47:17.585072 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0925 22:47:17.585076 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:47:17.585185 (+ 109us) server_negotiation.cc:407] Connection header received
0925 22:47:17.585674 (+ 489us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:47:17.585677 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:47:17.585725 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:47:17.585796 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:47:17.586129 (+ 333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:47:17.586876 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:47:17.588453 (+ 1577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:47:17.589240 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:47:17.589414 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:47:17.589487 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0925 22:47:17.589717 (+ 230us) server_negotiation.cc:299] Negotiation successful
0925 22:47:17.589828 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260925 22:47:17.589955 25271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:47:17.584521 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.198.1:51477 (local address 127.10.198.2:35091)
0925 22:47:17.584672 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0925 22:47:17.584678 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 22:47:17.585092 (+ 414us) server_negotiation.cc:407] Connection header received
0925 22:47:17.585369 (+ 277us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:47:17.585372 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:47:17.585432 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:47:17.585530 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:47:17.585882 (+ 352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:47:17.586562 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:47:17.588418 (+ 1856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:47:17.589125 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:47:17.589279 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:47:17.589338 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0925 22:47:17.589693 (+ 355us) server_negotiation.cc:299] Negotiation successful
0925 22:47:17.589792 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260925 22:47:17.589998 25285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:47:17.584411 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.198.2:35091 (local address 127.10.198.1:51477)
0925 22:47:17.584971 (+ 560us) negotiation.cc:107] Waiting for socket to connect
0925 22:47:17.584991 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0925 22:47:17.585229 (+ 238us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:47:17.585558 (+ 329us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:47:17.585561 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:47:17.585579 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:47:17.585699 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:47:17.585705 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:47:17.586703 (+ 998us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:47:17.586705 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:47:17.588226 (+ 1521us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:47:17.588240 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:47:17.589340 (+ 1100us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:47:17.589348 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:47:17.589480 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:47:17.589523 (+ 43us) client_negotiation.cc:769] Sending connection context
0925 22:47:17.589664 (+ 141us) client_negotiation.cc:240] Negotiation successful
0925 22:47:17.589783 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":436,"spinlock_wait_cycles":20608,"thread_start_us":171,"threads_started":1}
I20260925 22:47:17.590461 25286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:47:17.584445 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.198.3:37747 (local address 127.10.198.1:55511)
0925 22:47:17.585087 (+ 642us) negotiation.cc:107] Waiting for socket to connect
0925 22:47:17.585102 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 22:47:17.585229 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:47:17.585876 (+ 647us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:47:17.585879 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:47:17.585898 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:47:17.586010 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:47:17.586017 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:47:17.587025 (+ 1008us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:47:17.587029 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:47:17.588273 (+ 1244us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:47:17.588284 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:47:17.589432 (+ 1148us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:47:17.589436 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:47:17.589555 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:47:17.589579 (+ 24us) client_negotiation.cc:769] Sending connection context
0925 22:47:17.590208 (+ 629us) client_negotiation.cc:240] Negotiation successful
0925 22:47:17.590318 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":563,"spinlock_wait_cycles":13696,"thread_start_us":133,"threads_started":1}
I20260925 22:47:17.591142 25079 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6ec8f4e57b94888b44230fa02d0c216" candidate_uuid: "d446d20efa5e403c9ca66ad9e6bd6829" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e2485ed683004618a230d0a403a4d008" is_pre_election: true
I20260925 22:47:17.591361 25213 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6ec8f4e57b94888b44230fa02d0c216" candidate_uuid: "d446d20efa5e403c9ca66ad9e6bd6829" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e6fad9ce2c614cb08498cd3cdf453424" is_pre_election: true
I20260925 22:47:17.591604 25079 raft_consensus.cc:2454] T d6ec8f4e57b94888b44230fa02d0c216 P e2485ed683004618a230d0a403a4d008 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d446d20efa5e403c9ca66ad9e6bd6829 in term 0.
I20260925 22:47:17.591660 25213 raft_consensus.cc:2454] T d6ec8f4e57b94888b44230fa02d0c216 P e6fad9ce2c614cb08498cd3cdf453424 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d446d20efa5e403c9ca66ad9e6bd6829 in term 0.
I20260925 22:47:17.592324 24880 leader_election.cc:302] T d6ec8f4e57b94888b44230fa02d0c216 P d446d20efa5e403c9ca66ad9e6bd6829 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: d446d20efa5e403c9ca66ad9e6bd6829, e2485ed683004618a230d0a403a4d008, e6fad9ce2c614cb08498cd3cdf453424; no voters:
I20260925 22:47:17.592730 25282 raft_consensus.cc:2781] T d6ec8f4e57b94888b44230fa02d0c216 P d446d20efa5e403c9ca66ad9e6bd6829 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 22:47:17.592828 25282 raft_consensus.cc:482] T d6ec8f4e57b94888b44230fa02d0c216 P d446d20efa5e403c9ca66ad9e6bd6829 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 22:47:17.592859 25282 raft_consensus.cc:3037] T d6ec8f4e57b94888b44230fa02d0c216 P d446d20efa5e403c9ca66ad9e6bd6829 [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:47:17.594019 25282 raft_consensus.cc:504] T d6ec8f4e57b94888b44230fa02d0c216 P d446d20efa5e403c9ca66ad9e6bd6829 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2485ed683004618a230d0a403a4d008" member_type: VOTER last_known_addr { host: "127.10.198.2" port: 35091 } } peers { permanent_uuid: "d446d20efa5e403c9ca66ad9e6bd6829" member_type: VOTER last_known_addr { host: "127.10.198.1" port: 35361 } } peers { permanent_uuid: "e6fad9ce2c614cb08498cd3cdf453424" member_type: VOTER last_known_addr { host: "127.10.198.3" port: 37747 } }
I20260925 22:47:17.594836 25282 leader_election.cc:288] T d6ec8f4e57b94888b44230fa02d0c216 P d446d20efa5e403c9ca66ad9e6bd6829 [CANDIDATE]: Term 1 election: Requested vote from peers e2485ed683004618a230d0a403a4d008 (127.10.198.2:35091), e6fad9ce2c614cb08498cd3cdf453424 (127.10.198.3:37747)
I20260925 22:47:17.595395 25213 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6ec8f4e57b94888b44230fa02d0c216" candidate_uuid: "d446d20efa5e403c9ca66ad9e6bd6829" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e6fad9ce2c614cb08498cd3cdf453424"
I20260925 22:47:17.595395 25079 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6ec8f4e57b94888b44230fa02d0c216" candidate_uuid: "d446d20efa5e403c9ca66ad9e6bd6829" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e2485ed683004618a230d0a403a4d008"
I20260925 22:47:17.595571 25079 raft_consensus.cc:3037] T d6ec8f4e57b94888b44230fa02d0c216 P e2485ed683004618a230d0a403a4d008 [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:47:17.595571 25213 raft_consensus.cc:3037] T d6ec8f4e57b94888b44230fa02d0c216 P e6fad9ce2c614cb08498cd3cdf453424 [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:47:17.597126 25213 raft_consensus.cc:2454] T d6ec8f4e57b94888b44230fa02d0c216 P e6fad9ce2c614cb08498cd3cdf453424 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d446d20efa5e403c9ca66ad9e6bd6829 in term 1.
I20260925 22:47:17.597126 25079 raft_consensus.cc:2454] T d6ec8f4e57b94888b44230fa02d0c216 P e2485ed683004618a230d0a403a4d008 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d446d20efa5e403c9ca66ad9e6bd6829 in term 1.
I20260925 22:47:17.597673 24879 leader_election.cc:302] T d6ec8f4e57b94888b44230fa02d0c216 P d446d20efa5e403c9ca66ad9e6bd6829 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: d446d20efa5e403c9ca66ad9e6bd6829, e2485ed683004618a230d0a403a4d008, e6fad9ce2c614cb08498cd3cdf453424; no voters:
I20260925 22:47:17.598001 25282 raft_consensus.cc:2781] T d6ec8f4e57b94888b44230fa02d0c216 P d446d20efa5e403c9ca66ad9e6bd6829 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 22:47:17.598351 25282 raft_consensus.cc:686] T d6ec8f4e57b94888b44230fa02d0c216 P d446d20efa5e403c9ca66ad9e6bd6829 [term 1 LEADER]: Becoming Leader. State: Replica: d446d20efa5e403c9ca66ad9e6bd6829, State: Running, Role: LEADER
I20260925 22:47:17.598795 25282 consensus_queue.cc:237] T d6ec8f4e57b94888b44230fa02d0c216 P d446d20efa5e403c9ca66ad9e6bd6829 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2485ed683004618a230d0a403a4d008" member_type: VOTER last_known_addr { host: "127.10.198.2" port: 35091 } } peers { permanent_uuid: "d446d20efa5e403c9ca66ad9e6bd6829" member_type: VOTER last_known_addr { host: "127.10.198.1" port: 35361 } } peers { permanent_uuid: "e6fad9ce2c614cb08498cd3cdf453424" member_type: VOTER last_known_addr { host: "127.10.198.3" port: 37747 } }
I20260925 22:47:17.603120 24797 catalog_manager.cc:5495] T d6ec8f4e57b94888b44230fa02d0c216 P d446d20efa5e403c9ca66ad9e6bd6829 reported cstate change: term changed from 0 to 1, leader changed from <none> to d446d20efa5e403c9ca66ad9e6bd6829 (127.10.198.1). New cstate: current_term: 1 leader_uuid: "d446d20efa5e403c9ca66ad9e6bd6829" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2485ed683004618a230d0a403a4d008" member_type: VOTER last_known_addr { host: "127.10.198.2" port: 35091 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d446d20efa5e403c9ca66ad9e6bd6829" member_type: VOTER last_known_addr { host: "127.10.198.1" port: 35361 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e6fad9ce2c614cb08498cd3cdf453424" member_type: VOTER last_known_addr { host: "127.10.198.3" port: 37747 } health_report { overall_health: UNKNOWN } } }
I20260925 22:47:17.632807 25275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:47:17.628722 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39380 (local address 127.10.198.1:35361)
0925 22:47:17.629000 (+ 278us) server_negotiation.cc:206] Beginning negotiation
0925 22:47:17.629004 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:47:17.629022 (+ 18us) server_negotiation.cc:407] Connection header received
0925 22:47:17.629218 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:47:17.629221 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:47:17.629283 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:47:17.629406 (+ 123us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 22:47:17.629693 (+ 287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:47:17.630226 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:47:17.631151 (+ 925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:47:17.631387 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:47:17.631511 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:47:17.631888 (+ 377us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 22:47:17.632362 (+ 474us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 22:47:17.632513 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0925 22:47:17.632569 (+ 56us) server_negotiation.cc:299] Negotiation successful
0925 22:47:17.632639 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":182}
I20260925 22:47:17.640125 25213 raft_consensus.cc:1261] T d6ec8f4e57b94888b44230fa02d0c216 P e6fad9ce2c614cb08498cd3cdf453424 [term 1 FOLLOWER]: Refusing update from remote peer d446d20efa5e403c9ca66ad9e6bd6829: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 22:47:17.640154 25079 raft_consensus.cc:1261] T d6ec8f4e57b94888b44230fa02d0c216 P e2485ed683004618a230d0a403a4d008 [term 1 FOLLOWER]: Refusing update from remote peer d446d20efa5e403c9ca66ad9e6bd6829: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 22:47:17.641109 25287 consensus_queue.cc:1035] T d6ec8f4e57b94888b44230fa02d0c216 P d446d20efa5e403c9ca66ad9e6bd6829 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e6fad9ce2c614cb08498cd3cdf453424" member_type: VOTER last_known_addr { host: "127.10.198.3" port: 37747 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 22:47:17.641373 25282 consensus_queue.cc:1035] T d6ec8f4e57b94888b44230fa02d0c216 P d446d20efa5e403c9ca66ad9e6bd6829 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2485ed683004618a230d0a403a4d008" member_type: VOTER last_known_addr { host: "127.10.198.2" port: 35091 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260925 22:47:17.648420 25126 tablet.cc:2369] T d6ec8f4e57b94888b44230fa02d0c216 P e2485ed683004618a230d0a403a4d008: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 22:47:17.661819 24797 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38602:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 22:47:17.662092 24797 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38602:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 22:47:17.665398 24797 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 27eb88732a4e4caaae541c787b446b72: Sending DeleteTablet for 3 replicas of tablet d6ec8f4e57b94888b44230fa02d0c216
I20260925 22:47:17.666473 11032 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrWXm8X/build/debug/bin/kudu with pid 24862
I20260925 22:47:17.666462 25059 tablet_service.cc:1514] Processing DeleteTablet for tablet d6ec8f4e57b94888b44230fa02d0c216 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 22:47:17 UTC) from {username='kudu', principal='kudu/127.10.198.62@KRBTEST.COM'} at 127.0.0.1:36942
I20260925 22:47:17.666517 25193 tablet_service.cc:1514] Processing DeleteTablet for tablet d6ec8f4e57b94888b44230fa02d0c216 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 22:47:17 UTC) from {username='kudu', principal='kudu/127.10.198.62@KRBTEST.COM'} at 127.0.0.1:42292
I20260925 22:47:17.667431 25300 tablet_replica.cc:330] T d6ec8f4e57b94888b44230fa02d0c216 P e2485ed683004618a230d0a403a4d008: stopping tablet replica
I20260925 22:47:17.667977 25300 raft_consensus.cc:2229] T d6ec8f4e57b94888b44230fa02d0c216 P e2485ed683004618a230d0a403a4d008 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 22:47:17.668068 25301 tablet_replica.cc:330] T d6ec8f4e57b94888b44230fa02d0c216 P e6fad9ce2c614cb08498cd3cdf453424: stopping tablet replica
I20260925 22:47:17.668413 25301 raft_consensus.cc:2229] T d6ec8f4e57b94888b44230fa02d0c216 P e6fad9ce2c614cb08498cd3cdf453424 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 22:47:17.668448 25300 raft_consensus.cc:2258] T d6ec8f4e57b94888b44230fa02d0c216 P e2485ed683004618a230d0a403a4d008 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 22:47:17.668754 25301 raft_consensus.cc:2258] T d6ec8f4e57b94888b44230fa02d0c216 P e6fad9ce2c614cb08498cd3cdf453424 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 22:47:17.670334 25300 ts_tablet_manager.cc:1905] T d6ec8f4e57b94888b44230fa02d0c216 P e2485ed683004618a230d0a403a4d008: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 22:47:17.670372 25301 ts_tablet_manager.cc:1905] T d6ec8f4e57b94888b44230fa02d0c216 P e6fad9ce2c614cb08498cd3cdf453424: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 22:47:17.673240 25300 ts_tablet_manager.cc:1918] T d6ec8f4e57b94888b44230fa02d0c216 P e2485ed683004618a230d0a403a4d008: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 22:47:17.673390 25300 log.cc:1196] T d6ec8f4e57b94888b44230fa02d0c216 P e2485ed683004618a230d0a403a4d008: Deleting WAL directory at /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376259650642-11032-0/minicluster-data/ts-1/wal/wals/d6ec8f4e57b94888b44230fa02d0c216
I20260925 22:47:17.673678 25300 ts_tablet_manager.cc:1939] T d6ec8f4e57b94888b44230fa02d0c216 P e2485ed683004618a230d0a403a4d008: Deleting consensus metadata
I20260925 22:47:17.674649 25301 ts_tablet_manager.cc:1918] T d6ec8f4e57b94888b44230fa02d0c216 P e6fad9ce2c614cb08498cd3cdf453424: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 22:47:17.674863 25301 log.cc:1196] T d6ec8f4e57b94888b44230fa02d0c216 P e6fad9ce2c614cb08498cd3cdf453424: Deleting WAL directory at /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376259650642-11032-0/minicluster-data/ts-2/wal/wals/d6ec8f4e57b94888b44230fa02d0c216
W20260925 22:47:17.675187 24785 connection.cc:535] client connection to 127.10.198.1:35361 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260925 22:47:17.675311 25301 ts_tablet_manager.cc:1939] T d6ec8f4e57b94888b44230fa02d0c216 P e6fad9ce2c614cb08498cd3cdf453424: Deleting consensus metadata
W20260925 22:47:17.675380 24785 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260925 22:47:17.675944 24785 catalog_manager.cc:4841] TS e2485ed683004618a230d0a403a4d008 (127.10.198.2:35091): tablet d6ec8f4e57b94888b44230fa02d0c216 (table test-table [id=f1f12203d292477cac29d7fff1cedb52]) successfully deleted
I20260925 22:47:17.676473 11032 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrWXm8X/build/debug/bin/kudu with pid 24996
I20260925 22:47:17.676656 25270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:47:17.676306 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.198.1:35361 (local address 127.0.0.1:39392)
0925 22:47:17.676499 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0925 22:47:17.676574 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.198.1:35361: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112}
W20260925 22:47:17.677139 24785 catalog_manager.cc:4584] TS d446d20efa5e403c9ca66ad9e6bd6829 (127.10.198.1:35361): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet d6ec8f4e57b94888b44230fa02d0c216: Network error: Client connection negotiation failed: client connection to 127.10.198.1:35361: connect: Connection refused (error 111)
I20260925 22:47:17.678061 24786 catalog_manager.cc:4841] TS e6fad9ce2c614cb08498cd3cdf453424 (127.10.198.3:37747): tablet d6ec8f4e57b94888b44230fa02d0c216 (table test-table [id=f1f12203d292477cac29d7fff1cedb52]) successfully deleted
I20260925 22:47:17.683470 11032 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrWXm8X/build/debug/bin/kudu with pid 25130
I20260925 22:47:17.690359 11032 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrWXm8X/build/debug/bin/kudu with pid 24768
I20260925 22:47:17.697528 11032 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrWXm8X/build/debug/bin/kudu
/tmp/dist-test-taskrWXm8X/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376259650642-11032-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376259650642-11032-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376259650642-11032-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376259650642-11032-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.198.62:42687
--webserver_interface=127.10.198.62
--webserver_port=36809
--keytab_file=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376259650642-11032-0/krb5kdc/kudu_127.10.198.62.keytab
--principal=kudu/127.10.198.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.198.20:44287
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.198.62:42687
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376259650642-11032-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376259650642-11032-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376259650642-11032-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:47:17.826508 25303 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:47:17.826879 25303 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:47:17.826927 25303 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:47:17.831461 25303 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 22:47:17.831575 25303 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:47:17.831598 25303 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:47:17.831620 25303 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 22:47:17.831640 25303 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 22:47:17.837994 25303 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.198.20:44287
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376259650642-11032-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376259650642-11032-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.10.198.62:42687
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376259650642-11032-0/krb5kdc/kudu_127.10.198.62.keytab
--principal=kudu/127.10.198.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.198.62:42687
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376259650642-11032-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.198.62
--webserver_port=36809
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25303
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376259650642-11032-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:09:16 UTC on ccycloud.quasar-afmhyw.root.comops.site
build id 4673379
I20260925 22:47:17.839344 25303 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:47:17.840612 25303 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:47:17.857020 25310 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:47:17.856997 25312 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:47:17.856997 25309 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:47:17.857146 25303 server_base.cc:1034] running on GCE node
I20260925 22:47:17.858038 25303 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:47:17.859460 25303 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:47:17.860709 25303 hybrid_clock.cc:648] HybridClock initialized: now 1790376437860653 us; error 82 us; skew 500 ppm
Sep 25 22:47:17 dist-test-slave-5vdr krb5kdc[24755](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376437, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.198.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:47:17.864220 25303 init.cc:377] Logged in from keytab as kudu/127.10.198.62@KRBTEST.COM (short username kudu)
I20260925 22:47:17.865520 25303 webserver.cc:460] Webserver started at http://127.10.198.62:36809/ using document root <none> and password file <none>
I20260925 22:47:17.866190 25303 fs_manager.cc:359] Metadata directory not provided
I20260925 22:47:17.866283 25303 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:47:17.871135 25303 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.000s sys 0.004s
I20260925 22:47:17.873989 25319 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:47:17.875900 25303 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260925 22:47:17.876113 25303 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376259650642-11032-0/minicluster-data/master-0/data,/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376259650642-11032-0/minicluster-data/master-0/wal
uuid: "27eb88732a4e4caaae541c787b446b72"
format_stamp: "Formatted at 2026-09-25 22:47:16 on dist-test-slave-5vdr"
server_key: "e87b98d930f8e15835be623fbc0bd542"
server_key_iv: "802d4f5a06d643fdbf4e2c978d2ee152"
server_key_version: "encryptionkey@0"
I20260925 22:47:17.876878 25303 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376259650642-11032-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376259650642-11032-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376259650642-11032-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:47:17.895603 25303 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:47:17.899187 25303 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:47:17.899439 25303 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:47:17.908057 25303 rpc_server.cc:307] RPC server started. Bound to: 127.10.198.62:42687
I20260925 22:47:17.908351 25371 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.198.62:42687 every 8 connection(s)
I20260925 22:47:17.909447 25303 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376259650642-11032-0/minicluster-data/master-0/data/info.pb
I20260925 22:47:17.914562 25372 sys_catalog.cc:263] Verifying existing consensus state
I20260925 22:47:17.915975 11032 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrWXm8X/build/debug/bin/kudu as pid 25303
I20260925 22:47:17.916450 11032 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrWXm8X/build/debug/bin/kudu
/tmp/dist-test-taskrWXm8X/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376259650642-11032-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376259650642-11032-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376259650642-11032-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376259650642-11032-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.198.1:35361
--local_ip_for_outbound_sockets=127.10.198.1
--tserver_master_addrs=127.10.198.62:42687
--webserver_port=35377
--webserver_interface=127.10.198.1
--keytab_file=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376259650642-11032-0/krb5kdc/kudu_127.10.198.1.keytab
--principal=kudu/127.10.198.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.198.20:44287
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376259650642-11032-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376259650642-11032-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376259650642-11032-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 22:47:17.917085 25372 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 27eb88732a4e4caaae541c787b446b72: Bootstrap starting.
I20260925 22:47:17.942214 25372 log.cc:824] T 00000000000000000000000000000000 P 27eb88732a4e4caaae541c787b446b72: Log is configured to *not* fsync() on all Append() calls
I20260925 22:47:17.955327 25372 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 27eb88732a4e4caaae541c787b446b72: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 22:47:17.955864 25372 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 27eb88732a4e4caaae541c787b446b72: Bootstrap complete.
I20260925 22:47:17.961615 25372 raft_consensus.cc:348] T 00000000000000000000000000000000 P 27eb88732a4e4caaae541c787b446b72 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "27eb88732a4e4caaae541c787b446b72" member_type: VOTER last_known_addr { host: "127.10.198.62" port: 42687 } }
I20260925 22:47:17.962257 25372 raft_consensus.cc:729] T 00000000000000000000000000000000 P 27eb88732a4e4caaae541c787b446b72 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 27eb88732a4e4caaae541c787b446b72, State: Initialized, Role: FOLLOWER
I20260925 22:47:17.962949 25372 consensus_queue.cc:260] T 00000000000000000000000000000000 P 27eb88732a4e4caaae541c787b446b72 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "27eb88732a4e4caaae541c787b446b72" member_type: VOTER last_known_addr { host: "127.10.198.62" port: 42687 } }
I20260925 22:47:17.963158 25372 raft_consensus.cc:388] T 00000000000000000000000000000000 P 27eb88732a4e4caaae541c787b446b72 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 22:47:17.963232 25372 raft_consensus.cc:482] T 00000000000000000000000000000000 P 27eb88732a4e4caaae541c787b446b72 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 22:47:17.963339 25372 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 27eb88732a4e4caaae541c787b446b72 [term 1 FOLLOWER]: Advancing to term 2
I20260925 22:47:17.966318 25372 raft_consensus.cc:504] T 00000000000000000000000000000000 P 27eb88732a4e4caaae541c787b446b72 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "27eb88732a4e4caaae541c787b446b72" member_type: VOTER last_known_addr { host: "127.10.198.62" port: 42687 } }
I20260925 22:47:17.966799 25372 leader_election.cc:302] T 00000000000000000000000000000000 P 27eb88732a4e4caaae541c787b446b72 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 27eb88732a4e4caaae541c787b446b72; no voters:
I20260925 22:47:17.967199 25372 leader_election.cc:288] T 00000000000000000000000000000000 P 27eb88732a4e4caaae541c787b446b72 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260925 22:47:17.967417 25376 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 27eb88732a4e4caaae541c787b446b72 [term 2 FOLLOWER]: Leader election won for term 2
I20260925 22:47:17.967742 25376 raft_consensus.cc:686] T 00000000000000000000000000000000 P 27eb88732a4e4caaae541c787b446b72 [term 2 LEADER]: Becoming Leader. State: Replica: 27eb88732a4e4caaae541c787b446b72, State: Running, Role: LEADER
I20260925 22:47:17.968173 25376 consensus_queue.cc:237] T 00000000000000000000000000000000 P 27eb88732a4e4caaae541c787b446b72 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "27eb88732a4e4caaae541c787b446b72" member_type: VOTER last_known_addr { host: "127.10.198.62" port: 42687 } }
I20260925 22:47:17.968674 25372 sys_catalog.cc:564] T 00000000000000000000000000000000 P 27eb88732a4e4caaae541c787b446b72 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 22:47:17.970089 25377 sys_catalog.cc:455] T 00000000000000000000000000000000 P 27eb88732a4e4caaae541c787b446b72 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "27eb88732a4e4caaae541c787b446b72" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "27eb88732a4e4caaae541c787b446b72" member_type: VOTER last_known_addr { host: "127.10.198.62" port: 42687 } } }
I20260925 22:47:17.970098 25378 sys_catalog.cc:455] T 00000000000000000000000000000000 P 27eb88732a4e4caaae541c787b446b72 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 27eb88732a4e4caaae541c787b446b72. Latest consensus state: current_term: 2 leader_uuid: "27eb88732a4e4caaae541c787b446b72" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "27eb88732a4e4caaae541c787b446b72" member_type: VOTER last_known_addr { host: "127.10.198.62" port: 42687 } } }
I20260925 22:47:17.970279 25378 sys_catalog.cc:458] T 00000000000000000000000000000000 P 27eb88732a4e4caaae541c787b446b72 [sys.catalog]: This master's current role is: LEADER
I20260925 22:47:17.970287 25377 sys_catalog.cc:458] T 00000000000000000000000000000000 P 27eb88732a4e4caaae541c787b446b72 [sys.catalog]: This master's current role is: LEADER
I20260925 22:47:17.970834 25387 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 22:47:17.976115 25387 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 22:47:17.976861 25387 catalog_manager.cc:1259] Loaded cluster ID: f4e6616644f449c08dd601a6389da1b3
I20260925 22:47:17.976967 25387 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 22:47:17.978269 25387 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260925 22:47:17.981803 25387 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260925 22:47:17.987795 25387 catalog_manager.cc:1504] Loading token signing keys...
I20260925 22:47:17.988710 25387 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 27eb88732a4e4caaae541c787b446b72: Loaded TSK: 0
I20260925 22:47:17.989840 25387 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260925 22:47:18.075357 25374 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:47:18.075716 25374 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:47:18.075819 25374 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:47:18.080269 25374 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:47:18.080379 25374 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:47:18.080482 25374 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.198.1
I20260925 22:47:18.087072 25374 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.198.20:44287
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376259650642-11032-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376259650642-11032-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376259650642-11032-0/krb5kdc/kudu_127.10.198.1.keytab
--principal=kudu/127.10.198.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.198.1:35361
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376259650642-11032-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.198.1
--webserver_port=35377
--tserver_master_addrs=127.10.198.62:42687
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25374
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.198.1
--log_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376259650642-11032-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:09:16 UTC on ccycloud.quasar-afmhyw.root.comops.site
build id 4673379
I20260925 22:47:18.088353 25374 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:47:18.089538 25374 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:47:18.101384 25400 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:47:18.101433 25402 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:47:18.101781 25399 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:47:18.101781 25374 server_base.cc:1034] running on GCE node
I20260925 22:47:18.102360 25374 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:47:18.103163 25374 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:47:18.104430 25374 hybrid_clock.cc:648] HybridClock initialized: now 1790376438104398 us; error 57 us; skew 500 ppm
Sep 25 22:47:18 dist-test-slave-5vdr krb5kdc[24755](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376438, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.198.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:47:18.107673 25374 init.cc:377] Logged in from keytab as kudu/127.10.198.1@KRBTEST.COM (short username kudu)
I20260925 22:47:18.108961 25374 webserver.cc:460] Webserver started at http://127.10.198.1:35377/ using document root <none> and password file <none>
I20260925 22:47:18.109645 25374 fs_manager.cc:359] Metadata directory not provided
I20260925 22:47:18.109738 25374 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:47:18.114420 25374 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260925 22:47:18.117128 25409 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:47:18.118644 25374 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 22:47:18.118779 25374 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376259650642-11032-0/minicluster-data/ts-0/data,/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376259650642-11032-0/minicluster-data/ts-0/wal
uuid: "d446d20efa5e403c9ca66ad9e6bd6829"
format_stamp: "Formatted at 2026-09-25 22:47:16 on dist-test-slave-5vdr"
server_key: "d2f2d829885eb12f22e8b0b03c4559fa"
server_key_iv: "39ca185db2e434a48e3305f4d90043b9"
server_key_version: "encryptionkey@0"
I20260925 22:47:18.119445 25374 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376259650642-11032-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376259650642-11032-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376259650642-11032-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:47:18.152832 25374 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:47:18.156430 25374 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:47:18.156667 25374 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:47:18.157357 25374 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 22:47:18.158627 25416 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260925 22:47:18.163411 25374 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260925 22:47:18.163527 25374 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.005s user 0.001s sys 0.000s
I20260925 22:47:18.163625 25374 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260925 22:47:18.165755 25374 ts_tablet_manager.cc:610] Registered 1 tablets
I20260925 22:47:18.165820 25374 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.001s sys 0.000s
I20260925 22:47:18.166257 25416 tablet_bootstrap.cc:492] T d6ec8f4e57b94888b44230fa02d0c216 P d446d20efa5e403c9ca66ad9e6bd6829: Bootstrap starting.
I20260925 22:47:18.177493 25374 rpc_server.cc:307] RPC server started. Bound to: 127.10.198.1:35361
I20260925 22:47:18.177591 25523 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.198.1:35361 every 8 connection(s)
I20260925 22:47:18.178649 25374 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376259650642-11032-0/minicluster-data/ts-0/data/info.pb
I20260925 22:47:18.186049 11032 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrWXm8X/build/debug/bin/kudu as pid 25374
I20260925 22:47:18.186566 11032 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrWXm8X/build/debug/bin/kudu
/tmp/dist-test-taskrWXm8X/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376259650642-11032-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376259650642-11032-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376259650642-11032-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376259650642-11032-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.198.2:35091
--local_ip_for_outbound_sockets=127.10.198.2
--tserver_master_addrs=127.10.198.62:42687
--webserver_port=46157
--webserver_interface=127.10.198.2
--keytab_file=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376259650642-11032-0/krb5kdc/kudu_127.10.198.2.keytab
--principal=kudu/127.10.198.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.198.20:44287
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376259650642-11032-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376259650642-11032-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376259650642-11032-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 22:47:18.187429 25416 log.cc:824] T d6ec8f4e57b94888b44230fa02d0c216 P d446d20efa5e403c9ca66ad9e6bd6829: Log is configured to *not* fsync() on all Append() calls
Sep 25 22:47:18 dist-test-slave-5vdr krb5kdc[24755](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376438, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.198.1@KRBTEST.COM for kudu/127.10.198.62@KRBTEST.COM
I20260925 22:47:18.197014 25416 tablet_bootstrap.cc:492] T d6ec8f4e57b94888b44230fa02d0c216 P d446d20efa5e403c9ca66ad9e6bd6829: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 22:47:18.197527 25416 tablet_bootstrap.cc:492] T d6ec8f4e57b94888b44230fa02d0c216 P d446d20efa5e403c9ca66ad9e6bd6829: Bootstrap complete.
I20260925 22:47:18.198840 25416 ts_tablet_manager.cc:1397] T d6ec8f4e57b94888b44230fa02d0c216 P d446d20efa5e403c9ca66ad9e6bd6829: Time spent bootstrapping tablet: real 0.033s user 0.025s sys 0.004s
I20260925 22:47:18.201918 25526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:47:18.181022 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.198.1:46777 (local address 127.10.198.62:42687)
0925 22:47:18.181445 (+ 423us) server_negotiation.cc:206] Beginning negotiation
0925 22:47:18.181466 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0925 22:47:18.181963 (+ 497us) server_negotiation.cc:407] Connection header received
0925 22:47:18.183630 (+ 1667us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:47:18.183653 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:47:18.184131 (+ 478us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:47:18.184493 (+ 362us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:47:18.186371 (+ 1878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:47:18.187964 (+ 1593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:47:18.189845 (+ 1881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:47:18.190753 (+ 908us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:47:18.191019 (+ 266us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:47:18.194644 (+ 3625us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:47:18.194652 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:47:18.194670 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:47:18.194712 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:47:18.197388 (+ 2676us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:47:18.198328 (+ 940us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:47:18.198339 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:47:18.198348 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:47:18.198547 (+ 199us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:47:18.199099 (+ 552us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:47:18.199103 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:47:18.199105 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:47:18.199647 (+ 542us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:47:18.199788 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0925 22:47:18.200585 (+ 797us) server_negotiation.cc:299] Negotiation successful
0925 22:47:18.201048 (+ 463us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":256,"thread_start_us":130,"threads_started":1}
I20260925 22:47:18.201942 25527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:47:18.181064 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.198.62:42687 (local address 127.10.198.1:46777)
0925 22:47:18.181687 (+ 623us) negotiation.cc:107] Waiting for socket to connect
0925 22:47:18.181768 (+ 81us) client_negotiation.cc:174] Beginning negotiation
0925 22:47:18.183015 (+ 1247us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:47:18.184694 (+ 1679us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:47:18.184709 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:47:18.185311 (+ 602us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:47:18.186097 (+ 786us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:47:18.186122 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:47:18.188220 (+ 2098us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:47:18.188223 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:47:18.189634 (+ 1411us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:47:18.189653 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:47:18.190967 (+ 1314us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:47:18.190971 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:47:18.191162 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:47:18.192014 (+ 852us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:47:18.192037 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:47:18.194321 (+ 2284us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:47:18.197602 (+ 3281us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:47:18.197613 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:47:18.197631 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:47:18.198152 (+ 521us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:47:18.198700 (+ 548us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:47:18.198706 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:47:18.198711 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:47:18.198959 (+ 248us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:47:18.199807 (+ 848us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:47:18.199815 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:47:18.200197 (+ 382us) client_negotiation.cc:769] Sending connection context
0925 22:47:18.200432 (+ 235us) client_negotiation.cc:240] Negotiation successful
0925 22:47:18.200832 (+ 400us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":337,"thread_start_us":170,"threads_started":1}
I20260925 22:47:18.202064 25416 raft_consensus.cc:348] T d6ec8f4e57b94888b44230fa02d0c216 P d446d20efa5e403c9ca66ad9e6bd6829 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2485ed683004618a230d0a403a4d008" member_type: VOTER last_known_addr { host: "127.10.198.2" port: 35091 } } peers { permanent_uuid: "d446d20efa5e403c9ca66ad9e6bd6829" member_type: VOTER last_known_addr { host: "127.10.198.1" port: 35361 } } peers { permanent_uuid: "e6fad9ce2c614cb08498cd3cdf453424" member_type: VOTER last_known_addr { host: "127.10.198.3" port: 37747 } }
I20260925 22:47:18.202692 25416 raft_consensus.cc:729] T d6ec8f4e57b94888b44230fa02d0c216 P d446d20efa5e403c9ca66ad9e6bd6829 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: d446d20efa5e403c9ca66ad9e6bd6829, State: Initialized, Role: FOLLOWER
I20260925 22:47:18.203274 25416 consensus_queue.cc:260] T d6ec8f4e57b94888b44230fa02d0c216 P d446d20efa5e403c9ca66ad9e6bd6829 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2485ed683004618a230d0a403a4d008" member_type: VOTER last_known_addr { host: "127.10.198.2" port: 35091 } } peers { permanent_uuid: "d446d20efa5e403c9ca66ad9e6bd6829" member_type: VOTER last_known_addr { host: "127.10.198.1" port: 35361 } } peers { permanent_uuid: "e6fad9ce2c614cb08498cd3cdf453424" member_type: VOTER last_known_addr { host: "127.10.198.3" port: 37747 } }
I20260925 22:47:18.204547 25416 ts_tablet_manager.cc:1428] T d6ec8f4e57b94888b44230fa02d0c216 P d446d20efa5e403c9ca66ad9e6bd6829: Time spent starting tablet: real 0.006s user 0.008s sys 0.000s
I20260925 22:47:18.205345 25524 heartbeater.cc:347] Connected to a master server at 127.10.198.62:42687
I20260925 22:47:18.205607 25524 heartbeater.cc:464] Registering TS with master...
I20260925 22:47:18.206166 25524 heartbeater.cc:511] Master 127.10.198.62:42687 requested a full tablet report, sending...
I20260925 22:47:18.208993 25336 ts_manager.cc:194] Registered new tserver with Master: d446d20efa5e403c9ca66ad9e6bd6829 (127.10.198.1:35361)
I20260925 22:47:18.212714 25336 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.198.1@KRBTEST.COM'} at 127.10.198.1:46777
I20260925 22:47:18.215552 25524 heartbeater.cc:503] Master 127.10.198.62:42687 was elected leader, sending a full tablet report...
Sep 25 22:47:18 dist-test-slave-5vdr krb5kdc[24755](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376437, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.198.62@KRBTEST.COM for kudu/127.10.198.1@KRBTEST.COM
I20260925 22:47:18.226063 25531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:47:18.212243 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.198.1:35361 (local address 127.0.0.1:39394)
0925 22:47:18.213044 (+ 801us) negotiation.cc:107] Waiting for socket to connect
0925 22:47:18.213090 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0925 22:47:18.213254 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:47:18.213844 (+ 590us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:47:18.213851 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:47:18.214333 (+ 482us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:47:18.214488 (+ 155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:47:18.214497 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:47:18.215964 (+ 1467us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:47:18.215968 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:47:18.216796 (+ 828us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:47:18.216808 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:47:18.217481 (+ 673us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:47:18.217485 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:47:18.217614 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:47:18.218319 (+ 705us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:47:18.218334 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:47:18.220238 (+ 1904us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:47:18.223428 (+ 3190us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:47:18.223441 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:47:18.223446 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:47:18.223789 (+ 343us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:47:18.224371 (+ 582us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:47:18.224376 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:47:18.224378 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:47:18.224486 (+ 108us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:47:18.225216 (+ 730us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:47:18.225222 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:47:18.225354 (+ 132us) client_negotiation.cc:769] Sending connection context
0925 22:47:18.225543 (+ 189us) client_negotiation.cc:240] Negotiation successful
0925 22:47:18.225840 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":622,"thread_start_us":151,"threads_started":1}
I20260925 22:47:18.226063 25532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:47:18.212344 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39394 (local address 127.10.198.1:35361)
0925 22:47:18.212776 (+ 432us) server_negotiation.cc:206] Beginning negotiation
0925 22:47:18.212788 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 22:47:18.213233 (+ 445us) server_negotiation.cc:407] Connection header received
0925 22:47:18.213428 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:47:18.213434 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:47:18.213664 (+ 230us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:47:18.213828 (+ 164us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:47:18.214657 (+ 829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:47:18.215787 (+ 1130us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:47:18.216951 (+ 1164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:47:18.217330 (+ 379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:47:18.217467 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:47:18.220468 (+ 3001us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:47:18.220486 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:47:18.220501 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:47:18.220542 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:47:18.223215 (+ 2673us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:47:18.224069 (+ 854us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:47:18.224076 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:47:18.224077 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:47:18.224234 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:47:18.224607 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:47:18.224611 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:47:18.224613 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:47:18.224930 (+ 317us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:47:18.225161 (+ 231us) server_negotiation.cc:1035] Waiting for connection context
0925 22:47:18.225645 (+ 484us) server_negotiation.cc:299] Negotiation successful
0925 22:47:18.225854 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":322,"thread_start_us":107,"threads_started":1}
I20260925 22:47:18.227698 25456 tablet_service.cc:1514] Processing DeleteTablet for tablet d6ec8f4e57b94888b44230fa02d0c216 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 22:47:17 UTC) from {username='kudu', principal='kudu/127.10.198.62@KRBTEST.COM'} at 127.0.0.1:39394
I20260925 22:47:18.228974 25533 tablet_replica.cc:330] stopping tablet replica
I20260925 22:47:18.229360 25533 raft_consensus.cc:2229] T d6ec8f4e57b94888b44230fa02d0c216 P d446d20efa5e403c9ca66ad9e6bd6829 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 22:47:18.229689 25533 raft_consensus.cc:2258] T d6ec8f4e57b94888b44230fa02d0c216 P d446d20efa5e403c9ca66ad9e6bd6829 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 22:47:18.230907 25533 ts_tablet_manager.cc:1905] T d6ec8f4e57b94888b44230fa02d0c216 P d446d20efa5e403c9ca66ad9e6bd6829: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 22:47:18.236281 25533 ts_tablet_manager.cc:1918] T d6ec8f4e57b94888b44230fa02d0c216 P d446d20efa5e403c9ca66ad9e6bd6829: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 22:47:18.236454 25533 log.cc:1196] T d6ec8f4e57b94888b44230fa02d0c216 P d446d20efa5e403c9ca66ad9e6bd6829: Deleting WAL directory at /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376259650642-11032-0/minicluster-data/ts-0/wal/wals/d6ec8f4e57b94888b44230fa02d0c216
I20260925 22:47:18.236760 25533 ts_tablet_manager.cc:1939] T d6ec8f4e57b94888b44230fa02d0c216 P d446d20efa5e403c9ca66ad9e6bd6829: Deleting consensus metadata
I20260925 22:47:18.238783 25321 catalog_manager.cc:4841] TS d446d20efa5e403c9ca66ad9e6bd6829 (127.10.198.1:35361): tablet d6ec8f4e57b94888b44230fa02d0c216 (table test-table [id=f1f12203d292477cac29d7fff1cedb52]) successfully deleted
W20260925 22:47:18.318395 25528 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:47:18.318723 25528 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:47:18.318770 25528 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:47:18.323412 25528 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:47:18.323498 25528 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:47:18.323609 25528 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.198.2
I20260925 22:47:18.329805 25528 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.198.20:44287
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376259650642-11032-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376259650642-11032-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376259650642-11032-0/krb5kdc/kudu_127.10.198.2.keytab
--principal=kudu/127.10.198.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.198.2:35091
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376259650642-11032-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.198.2
--webserver_port=46157
--tserver_master_addrs=127.10.198.62:42687
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25528
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.198.2
--log_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376259650642-11032-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:09:16 UTC on ccycloud.quasar-afmhyw.root.comops.site
build id 4673379
I20260925 22:47:18.331318 25528 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:47:18.332628 25528 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:47:18.344161 25539 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:47:18.344131 25541 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:47:18.344408 25528 server_base.cc:1034] running on GCE node
W20260925 22:47:18.344575 25538 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:47:18.345045 25528 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:47:18.345815 25528 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:47:18.347127 25528 hybrid_clock.cc:648] HybridClock initialized: now 1790376438347098 us; error 44 us; skew 500 ppm
Sep 25 22:47:18 dist-test-slave-5vdr krb5kdc[24755](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376438, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.198.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:47:18.350852 25528 init.cc:377] Logged in from keytab as kudu/127.10.198.2@KRBTEST.COM (short username kudu)
I20260925 22:47:18.352123 25528 webserver.cc:460] Webserver started at http://127.10.198.2:46157/ using document root <none> and password file <none>
I20260925 22:47:18.352818 25528 fs_manager.cc:359] Metadata directory not provided
I20260925 22:47:18.352950 25528 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:47:18.357748 25528 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260925 22:47:18.360433 25548 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:47:18.361893 25528 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 22:47:18.362087 25528 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376259650642-11032-0/minicluster-data/ts-1/data,/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376259650642-11032-0/minicluster-data/ts-1/wal
uuid: "e2485ed683004618a230d0a403a4d008"
format_stamp: "Formatted at 2026-09-25 22:47:17 on dist-test-slave-5vdr"
server_key: "796a5bd9100cbda6ca63715963ec177c"
server_key_iv: "710b02d7903df43cd507750f807f85d8"
server_key_version: "encryptionkey@0"
I20260925 22:47:18.362761 25528 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376259650642-11032-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376259650642-11032-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376259650642-11032-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:47:18.377496 25528 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:47:18.381475 25528 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:47:18.381781 25528 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:47:18.382694 25528 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 22:47:18.383728 25528 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:47:18.383780 25528 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:47:18.383823 25528 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:47:18.383908 25528 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 22:47:18.394444 25528 rpc_server.cc:307] RPC server started. Bound to: 127.10.198.2:35091
I20260925 22:47:18.394613 25661 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.198.2:35091 every 8 connection(s)
I20260925 22:47:18.395689 25528 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376259650642-11032-0/minicluster-data/ts-1/data/info.pb
I20260925 22:47:18.405047 11032 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrWXm8X/build/debug/bin/kudu as pid 25528
I20260925 22:47:18.405529 11032 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrWXm8X/build/debug/bin/kudu
/tmp/dist-test-taskrWXm8X/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376259650642-11032-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376259650642-11032-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376259650642-11032-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376259650642-11032-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.198.3:37747
--local_ip_for_outbound_sockets=127.10.198.3
--tserver_master_addrs=127.10.198.62:42687
--webserver_port=42855
--webserver_interface=127.10.198.3
--keytab_file=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376259650642-11032-0/krb5kdc/kudu_127.10.198.3.keytab
--principal=kudu/127.10.198.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.198.20:44287
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376259650642-11032-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376259650642-11032-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376259650642-11032-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 25 22:47:18 dist-test-slave-5vdr krb5kdc[24755](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376438, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.198.2@KRBTEST.COM for kudu/127.10.198.62@KRBTEST.COM
I20260925 22:47:18.412348 25526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:47:18.397758 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.198.2:50441 (local address 127.10.198.62:42687)
0925 22:47:18.397927 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0925 22:47:18.397933 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 22:47:18.399009 (+ 1076us) server_negotiation.cc:407] Connection header received
0925 22:47:18.400620 (+ 1611us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:47:18.400624 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:47:18.400688 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:47:18.400773 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:47:18.402377 (+ 1604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:47:18.402947 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:47:18.404437 (+ 1490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:47:18.404772 (+ 335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:47:18.404919 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:47:18.408134 (+ 3215us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:47:18.408138 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:47:18.408140 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:47:18.408172 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:47:18.409846 (+ 1674us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:47:18.410540 (+ 694us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:47:18.410543 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:47:18.410545 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:47:18.410640 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:47:18.411052 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:47:18.411055 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:47:18.411057 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:47:18.411268 (+ 211us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:47:18.411384 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0925 22:47:18.412017 (+ 633us) server_negotiation.cc:299] Negotiation successful
0925 22:47:18.412176 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260925 22:47:18.413285 25664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:47:18.398259 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.198.62:42687 (local address 127.10.198.2:50441)
0925 22:47:18.398800 (+ 541us) negotiation.cc:107] Waiting for socket to connect
0925 22:47:18.398853 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0925 22:47:18.400338 (+ 1485us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:47:18.400985 (+ 647us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:47:18.400994 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:47:18.401453 (+ 459us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:47:18.402176 (+ 723us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:47:18.402197 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:47:18.403114 (+ 917us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:47:18.403118 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:47:18.404251 (+ 1133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:47:18.404265 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:47:18.404990 (+ 725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:47:18.404993 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:47:18.405175 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:47:18.406008 (+ 833us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:47:18.406029 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:47:18.407834 (+ 1805us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:47:18.410017 (+ 2183us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:47:18.410025 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:47:18.410048 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:47:18.410396 (+ 348us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:47:18.410748 (+ 352us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:47:18.410752 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:47:18.410754 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:47:18.410937 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:47:18.411388 (+ 451us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:47:18.411398 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:47:18.411701 (+ 303us) client_negotiation.cc:769] Sending connection context
0925 22:47:18.411962 (+ 261us) client_negotiation.cc:240] Negotiation successful
0925 22:47:18.412358 (+ 396us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":321,"thread_start_us":129,"threads_started":1}
I20260925 22:47:18.415675 25662 heartbeater.cc:347] Connected to a master server at 127.10.198.62:42687
I20260925 22:47:18.416076 25662 heartbeater.cc:464] Registering TS with master...
I20260925 22:47:18.416877 25662 heartbeater.cc:511] Master 127.10.198.62:42687 requested a full tablet report, sending...
I20260925 22:47:18.418381 25336 ts_manager.cc:194] Registered new tserver with Master: e2485ed683004618a230d0a403a4d008 (127.10.198.2:35091)
I20260925 22:47:18.419039 25336 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.198.2@KRBTEST.COM'} at 127.10.198.2:50441
W20260925 22:47:18.531572 25665 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:47:18.531944 25665 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:47:18.531994 25665 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:47:18.536520 25665 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:47:18.536620 25665 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:47:18.536690 25665 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.198.3
I20260925 22:47:18.542801 25665 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.198.20:44287
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376259650642-11032-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376259650642-11032-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376259650642-11032-0/krb5kdc/kudu_127.10.198.3.keytab
--principal=kudu/127.10.198.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.198.3:37747
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376259650642-11032-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.198.3
--webserver_port=42855
--tserver_master_addrs=127.10.198.62:42687
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25665
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.198.3
--log_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376259650642-11032-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:09:16 UTC on ccycloud.quasar-afmhyw.root.comops.site
build id 4673379
I20260925 22:47:18.544066 25665 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:47:18.545153 25665 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:47:18.556296 25670 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:47:18.556352 25665 server_base.cc:1034] running on GCE node
W20260925 22:47:18.556288 25673 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:47:18.556288 25671 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:47:18.556958 25665 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:47:18.557765 25665 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:47:18.558993 25665 hybrid_clock.cc:648] HybridClock initialized: now 1790376438558971 us; error 44 us; skew 500 ppm
Sep 25 22:47:18 dist-test-slave-5vdr krb5kdc[24755](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376438, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.198.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:47:18.562420 25665 init.cc:377] Logged in from keytab as kudu/127.10.198.3@KRBTEST.COM (short username kudu)
I20260925 22:47:18.563454 25665 webserver.cc:460] Webserver started at http://127.10.198.3:42855/ using document root <none> and password file <none>
I20260925 22:47:18.564167 25665 fs_manager.cc:359] Metadata directory not provided
I20260925 22:47:18.564270 25665 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:47:18.568908 25665 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260925 22:47:18.571559 25680 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:47:18.573235 25665 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 22:47:18.573407 25665 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376259650642-11032-0/minicluster-data/ts-2/data,/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376259650642-11032-0/minicluster-data/ts-2/wal
uuid: "e6fad9ce2c614cb08498cd3cdf453424"
format_stamp: "Formatted at 2026-09-25 22:47:17 on dist-test-slave-5vdr"
server_key: "1ebdbebba049bc1ce205275c6fad909d"
server_key_iv: "f7e50512d36c1fc08f3d67e35c186e1f"
server_key_version: "encryptionkey@0"
I20260925 22:47:18.574047 25665 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376259650642-11032-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376259650642-11032-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376259650642-11032-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:47:18.620185 25665 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:47:18.624420 25665 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:47:18.624675 25665 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:47:18.625723 25665 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 22:47:18.626816 25665 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:47:18.626873 25665 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:47:18.626951 25665 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:47:18.626983 25665 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 22:47:18.637187 25665 rpc_server.cc:307] RPC server started. Bound to: 127.10.198.3:37747
I20260925 22:47:18.637243 25793 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.198.3:37747 every 8 connection(s)
I20260925 22:47:18.638386 25665 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376259650642-11032-0/minicluster-data/ts-2/data/info.pb
I20260925 22:47:18.644654 11032 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrWXm8X/build/debug/bin/kudu as pid 25665
Sep 25 22:47:18 dist-test-slave-5vdr krb5kdc[24755](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376438, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.198.3@KRBTEST.COM for kudu/127.10.198.62@KRBTEST.COM
I20260925 22:47:18.655350 25798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:47:18.645539 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38616 (local address 127.10.198.62:42687)
0925 22:47:18.646011 (+ 472us) server_negotiation.cc:206] Beginning negotiation
0925 22:47:18.646015 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:47:18.646031 (+ 16us) server_negotiation.cc:407] Connection header received
0925 22:47:18.646073 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:47:18.646076 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:47:18.646130 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:47:18.646239 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:47:18.647037 (+ 798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:47:18.647747 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:47:18.648925 (+ 1178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:47:18.649324 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:47:18.649459 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:47:18.650993 (+ 1534us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:47:18.650999 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:47:18.651003 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:47:18.651050 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:47:18.653226 (+ 2176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:47:18.653821 (+ 595us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:47:18.653824 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:47:18.653826 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:47:18.653976 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:47:18.654332 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:47:18.654336 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:47:18.654351 (+ 15us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:47:18.654642 (+ 291us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:47:18.654877 (+ 235us) server_negotiation.cc:1035] Waiting for connection context
0925 22:47:18.655052 (+ 175us) server_negotiation.cc:299] Negotiation successful
0925 22:47:18.655174 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":379,"thread_start_us":81,"threads_started":1}
I20260925 22:47:18.655814 25526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:47:18.640559 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.198.3:36329 (local address 127.10.198.62:42687)
0925 22:47:18.640713 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0925 22:47:18.640717 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:47:18.641750 (+ 1033us) server_negotiation.cc:407] Connection header received
0925 22:47:18.643233 (+ 1483us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:47:18.643237 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:47:18.643299 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:47:18.643397 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:47:18.644966 (+ 1569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:47:18.645694 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:47:18.647547 (+ 1853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:47:18.647798 (+ 251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:47:18.647988 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:47:18.651308 (+ 3320us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:47:18.651311 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:47:18.651313 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:47:18.651350 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:47:18.653242 (+ 1892us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:47:18.653989 (+ 747us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:47:18.653992 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:47:18.653995 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:47:18.654086 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:47:18.654529 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:47:18.654532 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:47:18.654534 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:47:18.654733 (+ 199us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:47:18.654818 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0925 22:47:18.655546 (+ 728us) server_negotiation.cc:299] Negotiation successful
0925 22:47:18.655668 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":61,"spinlock_wait_cycles":5504}
I20260925 22:47:18.656975 25796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:47:18.641015 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.198.62:42687 (local address 127.10.198.3:36329)
0925 22:47:18.641545 (+ 530us) negotiation.cc:107] Waiting for socket to connect
0925 22:47:18.641598 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0925 22:47:18.642934 (+ 1336us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:47:18.643606 (+ 672us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:47:18.643614 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:47:18.644138 (+ 524us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:47:18.644723 (+ 585us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:47:18.644740 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:47:18.645880 (+ 1140us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:47:18.645884 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:47:18.647362 (+ 1478us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:47:18.647379 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:47:18.648039 (+ 660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:47:18.648043 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:47:18.648230 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:47:18.649007 (+ 777us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:47:18.649031 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:47:18.651091 (+ 2060us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:47:18.653398 (+ 2307us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:47:18.653405 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:47:18.653428 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:47:18.653851 (+ 423us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:47:18.654189 (+ 338us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:47:18.654192 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:47:18.654194 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:47:18.654408 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:47:18.654841 (+ 433us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:47:18.654851 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:47:18.655235 (+ 384us) client_negotiation.cc:769] Sending connection context
0925 22:47:18.655498 (+ 263us) client_negotiation.cc:240] Negotiation successful
0925 22:47:18.655923 (+ 425us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":318,"thread_start_us":156,"threads_started":1}
I20260925 22:47:18.658181 25794 heartbeater.cc:347] Connected to a master server at 127.10.198.62:42687
I20260925 22:47:18.658476 25794 heartbeater.cc:464] Registering TS with master...
I20260925 22:47:18.659070 25794 heartbeater.cc:511] Master 127.10.198.62:42687 requested a full tablet report, sending...
I20260925 22:47:18.660346 25336 ts_manager.cc:194] Registered new tserver with Master: e6fad9ce2c614cb08498cd3cdf453424 (127.10.198.3:37747)
I20260925 22:47:18.660972 25336 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.198.3@KRBTEST.COM'} at 127.10.198.3:36329
I20260925 22:47:18.669862 11032 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 22:47:18.675118 25526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:47:18.670638 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38626 (local address 127.10.198.62:42687)
0925 22:47:18.670790 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0925 22:47:18.670795 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:47:18.671045 (+ 250us) server_negotiation.cc:407] Connection header received
0925 22:47:18.671228 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:47:18.671231 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:47:18.671358 (+ 127us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:47:18.671466 (+ 108us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 22:47:18.671731 (+ 265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:47:18.672547 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:47:18.673518 (+ 971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:47:18.673757 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:47:18.673856 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:47:18.674185 (+ 329us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 22:47:18.674651 (+ 466us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 22:47:18.674747 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0925 22:47:18.674825 (+ 78us) server_negotiation.cc:299] Negotiation successful
0925 22:47:18.674880 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260925 22:47:18.676018 25336 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:38626:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 22:47:18.679574 25336 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 22:47:18.688812 25456 tablet_service.cc:1467] Processing CreateTablet for tablet ed68a1581d914821b50a865996d12512 (DEFAULT_TABLE table=test-table [id=2dc15b00d9af44489869d3c0c85c1d0c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 22:47:18.689785 25456 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ed68a1581d914821b50a865996d12512. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:47:18.693212 25416 tablet_bootstrap.cc:492] T ed68a1581d914821b50a865996d12512 P d446d20efa5e403c9ca66ad9e6bd6829: Bootstrap starting.
I20260925 22:47:18.694236 25416 tablet_bootstrap.cc:654] T ed68a1581d914821b50a865996d12512 P d446d20efa5e403c9ca66ad9e6bd6829: Neither blocks nor log segments found. Creating new log.
I20260925 22:47:18.694700 25531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:47:18.687615 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.198.2:35091 (local address 127.0.0.1:36948)
0925 22:47:18.687773 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0925 22:47:18.687791 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0925 22:47:18.687911 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:47:18.689319 (+ 1408us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:47:18.689322 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:47:18.689341 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:47:18.689478 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:47:18.689486 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:47:18.691490 (+ 2004us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:47:18.691495 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:47:18.692626 (+ 1131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:47:18.692641 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:47:18.694069 (+ 1428us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:47:18.694076 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:47:18.694227 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:47:18.694295 (+ 68us) client_negotiation.cc:769] Sending connection context
0925 22:47:18.694384 (+ 89us) client_negotiation.cc:240] Negotiation successful
0925 22:47:18.694513 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":67}
I20260925 22:47:18.695032 25804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:47:18.688112 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.198.3:37747 (local address 127.0.0.1:42308)
0925 22:47:18.688474 (+ 362us) negotiation.cc:107] Waiting for socket to connect
0925 22:47:18.688491 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 22:47:18.688588 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:47:18.689530 (+ 942us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:47:18.689534 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:47:18.689557 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:47:18.689656 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:47:18.689662 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:47:18.691628 (+ 1966us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:47:18.691635 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:47:18.692813 (+ 1178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:47:18.692829 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:47:18.694515 (+ 1686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:47:18.694519 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:47:18.694639 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:47:18.694669 (+ 30us) client_negotiation.cc:769] Sending connection context
0925 22:47:18.694768 (+ 99us) client_negotiation.cc:240] Negotiation successful
0925 22:47:18.694865 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":121,"threads_started":1}
I20260925 22:47:18.695109 25805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:47:18.688554 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42308 (local address 127.10.198.3:37747)
0925 22:47:18.688924 (+ 370us) server_negotiation.cc:206] Beginning negotiation
0925 22:47:18.688938 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0925 22:47:18.688965 (+ 27us) server_negotiation.cc:407] Connection header received
0925 22:47:18.689030 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:47:18.689034 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:47:18.689175 (+ 141us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:47:18.689819 (+ 644us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:47:18.690055 (+ 236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:47:18.691321 (+ 1266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:47:18.692982 (+ 1661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:47:18.694366 (+ 1384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:47:18.694511 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:47:18.694678 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0925 22:47:18.694805 (+ 127us) server_negotiation.cc:299] Negotiation successful
0925 22:47:18.694933 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":272,"thread_start_us":192,"threads_started":1}
I20260925 22:47:18.695531 25803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:47:18.687894 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36948 (local address 127.10.198.2:35091)
0925 22:47:18.688558 (+ 664us) server_negotiation.cc:206] Beginning negotiation
0925 22:47:18.688577 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0925 22:47:18.688602 (+ 25us) server_negotiation.cc:407] Connection header received
0925 22:47:18.688682 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:47:18.688688 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:47:18.688923 (+ 235us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:47:18.689663 (+ 740us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:47:18.689841 (+ 178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:47:18.691319 (+ 1478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:47:18.692816 (+ 1497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:47:18.693890 (+ 1074us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:47:18.694042 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:47:18.694247 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0925 22:47:18.695038 (+ 791us) server_negotiation.cc:299] Negotiation successful
0925 22:47:18.695152 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":560,"thread_start_us":172,"threads_started":1}
I20260925 22:47:18.696099 25416 tablet_bootstrap.cc:492] T ed68a1581d914821b50a865996d12512 P d446d20efa5e403c9ca66ad9e6bd6829: No bootstrap required, opened a new log
I20260925 22:47:18.696235 25416 ts_tablet_manager.cc:1397] T ed68a1581d914821b50a865996d12512 P d446d20efa5e403c9ca66ad9e6bd6829: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260925 22:47:18.696746 25416 raft_consensus.cc:348] T ed68a1581d914821b50a865996d12512 P d446d20efa5e403c9ca66ad9e6bd6829 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2485ed683004618a230d0a403a4d008" member_type: VOTER last_known_addr { host: "127.10.198.2" port: 35091 } } peers { permanent_uuid: "e6fad9ce2c614cb08498cd3cdf453424" member_type: VOTER last_known_addr { host: "127.10.198.3" port: 37747 } } peers { permanent_uuid: "d446d20efa5e403c9ca66ad9e6bd6829" member_type: VOTER last_known_addr { host: "127.10.198.1" port: 35361 } }
I20260925 22:47:18.696887 25416 raft_consensus.cc:374] T ed68a1581d914821b50a865996d12512 P d446d20efa5e403c9ca66ad9e6bd6829 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:47:18.696935 25416 raft_consensus.cc:729] T ed68a1581d914821b50a865996d12512 P d446d20efa5e403c9ca66ad9e6bd6829 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d446d20efa5e403c9ca66ad9e6bd6829, State: Initialized, Role: FOLLOWER
I20260925 22:47:18.697093 25416 consensus_queue.cc:260] T ed68a1581d914821b50a865996d12512 P d446d20efa5e403c9ca66ad9e6bd6829 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2485ed683004618a230d0a403a4d008" member_type: VOTER last_known_addr { host: "127.10.198.2" port: 35091 } } peers { permanent_uuid: "e6fad9ce2c614cb08498cd3cdf453424" member_type: VOTER last_known_addr { host: "127.10.198.3" port: 37747 } } peers { permanent_uuid: "d446d20efa5e403c9ca66ad9e6bd6829" member_type: VOTER last_known_addr { host: "127.10.198.1" port: 35361 } }
I20260925 22:47:18.697374 25416 ts_tablet_manager.cc:1428] T ed68a1581d914821b50a865996d12512 P d446d20efa5e403c9ca66ad9e6bd6829: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260925 22:47:18.699096 25728 tablet_service.cc:1467] Processing CreateTablet for tablet ed68a1581d914821b50a865996d12512 (DEFAULT_TABLE table=test-table [id=2dc15b00d9af44489869d3c0c85c1d0c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 22:47:18.700606 25728 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ed68a1581d914821b50a865996d12512. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:47:18.700809 25596 tablet_service.cc:1467] Processing CreateTablet for tablet ed68a1581d914821b50a865996d12512 (DEFAULT_TABLE table=test-table [id=2dc15b00d9af44489869d3c0c85c1d0c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 22:47:18.701996 25596 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ed68a1581d914821b50a865996d12512. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:47:18.709518 25807 tablet_bootstrap.cc:492] T ed68a1581d914821b50a865996d12512 P e2485ed683004618a230d0a403a4d008: Bootstrap starting.
I20260925 22:47:18.709522 25808 tablet_bootstrap.cc:492] T ed68a1581d914821b50a865996d12512 P e6fad9ce2c614cb08498cd3cdf453424: Bootstrap starting.
I20260925 22:47:18.712862 25808 tablet_bootstrap.cc:654] T ed68a1581d914821b50a865996d12512 P e6fad9ce2c614cb08498cd3cdf453424: Neither blocks nor log segments found. Creating new log.
I20260925 22:47:18.712949 25807 tablet_bootstrap.cc:654] T ed68a1581d914821b50a865996d12512 P e2485ed683004618a230d0a403a4d008: Neither blocks nor log segments found. Creating new log.
I20260925 22:47:18.713804 25807 log.cc:824] T ed68a1581d914821b50a865996d12512 P e2485ed683004618a230d0a403a4d008: Log is configured to *not* fsync() on all Append() calls
I20260925 22:47:18.713804 25808 log.cc:824] T ed68a1581d914821b50a865996d12512 P e6fad9ce2c614cb08498cd3cdf453424: Log is configured to *not* fsync() on all Append() calls
I20260925 22:47:18.716459 25808 tablet_bootstrap.cc:492] T ed68a1581d914821b50a865996d12512 P e6fad9ce2c614cb08498cd3cdf453424: No bootstrap required, opened a new log
I20260925 22:47:18.716769 25808 ts_tablet_manager.cc:1397] T ed68a1581d914821b50a865996d12512 P e6fad9ce2c614cb08498cd3cdf453424: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20260925 22:47:18.716668 25807 tablet_bootstrap.cc:492] T ed68a1581d914821b50a865996d12512 P e2485ed683004618a230d0a403a4d008: No bootstrap required, opened a new log
I20260925 22:47:18.717175 25807 ts_tablet_manager.cc:1397] T ed68a1581d914821b50a865996d12512 P e2485ed683004618a230d0a403a4d008: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260925 22:47:18.720449 25807 raft_consensus.cc:348] T ed68a1581d914821b50a865996d12512 P e2485ed683004618a230d0a403a4d008 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2485ed683004618a230d0a403a4d008" member_type: VOTER last_known_addr { host: "127.10.198.2" port: 35091 } } peers { permanent_uuid: "e6fad9ce2c614cb08498cd3cdf453424" member_type: VOTER last_known_addr { host: "127.10.198.3" port: 37747 } } peers { permanent_uuid: "d446d20efa5e403c9ca66ad9e6bd6829" member_type: VOTER last_known_addr { host: "127.10.198.1" port: 35361 } }
I20260925 22:47:18.720757 25807 raft_consensus.cc:374] T ed68a1581d914821b50a865996d12512 P e2485ed683004618a230d0a403a4d008 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:47:18.720804 25807 raft_consensus.cc:729] T ed68a1581d914821b50a865996d12512 P e2485ed683004618a230d0a403a4d008 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e2485ed683004618a230d0a403a4d008, State: Initialized, Role: FOLLOWER
I20260925 22:47:18.720579 25808 raft_consensus.cc:348] T ed68a1581d914821b50a865996d12512 P e6fad9ce2c614cb08498cd3cdf453424 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2485ed683004618a230d0a403a4d008" member_type: VOTER last_known_addr { host: "127.10.198.2" port: 35091 } } peers { permanent_uuid: "e6fad9ce2c614cb08498cd3cdf453424" member_type: VOTER last_known_addr { host: "127.10.198.3" port: 37747 } } peers { permanent_uuid: "d446d20efa5e403c9ca66ad9e6bd6829" member_type: VOTER last_known_addr { host: "127.10.198.1" port: 35361 } }
I20260925 22:47:18.720938 25808 raft_consensus.cc:374] T ed68a1581d914821b50a865996d12512 P e6fad9ce2c614cb08498cd3cdf453424 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:47:18.721028 25808 raft_consensus.cc:729] T ed68a1581d914821b50a865996d12512 P e6fad9ce2c614cb08498cd3cdf453424 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e6fad9ce2c614cb08498cd3cdf453424, State: Initialized, Role: FOLLOWER
I20260925 22:47:18.721411 25807 consensus_queue.cc:260] T ed68a1581d914821b50a865996d12512 P e2485ed683004618a230d0a403a4d008 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2485ed683004618a230d0a403a4d008" member_type: VOTER last_known_addr { host: "127.10.198.2" port: 35091 } } peers { permanent_uuid: "e6fad9ce2c614cb08498cd3cdf453424" member_type: VOTER last_known_addr { host: "127.10.198.3" port: 37747 } } peers { permanent_uuid: "d446d20efa5e403c9ca66ad9e6bd6829" member_type: VOTER last_known_addr { host: "127.10.198.1" port: 35361 } }
I20260925 22:47:18.721575 25808 consensus_queue.cc:260] T ed68a1581d914821b50a865996d12512 P e6fad9ce2c614cb08498cd3cdf453424 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2485ed683004618a230d0a403a4d008" member_type: VOTER last_known_addr { host: "127.10.198.2" port: 35091 } } peers { permanent_uuid: "e6fad9ce2c614cb08498cd3cdf453424" member_type: VOTER last_known_addr { host: "127.10.198.3" port: 37747 } } peers { permanent_uuid: "d446d20efa5e403c9ca66ad9e6bd6829" member_type: VOTER last_known_addr { host: "127.10.198.1" port: 35361 } }
I20260925 22:47:18.722352 25662 heartbeater.cc:503] Master 127.10.198.62:42687 was elected leader, sending a full tablet report...
I20260925 22:47:18.722467 25794 heartbeater.cc:503] Master 127.10.198.62:42687 was elected leader, sending a full tablet report...
I20260925 22:47:18.722867 25807 ts_tablet_manager.cc:1428] T ed68a1581d914821b50a865996d12512 P e2485ed683004618a230d0a403a4d008: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260925 22:47:18.722860 25808 ts_tablet_manager.cc:1428] T ed68a1581d914821b50a865996d12512 P e6fad9ce2c614cb08498cd3cdf453424: Time spent starting tablet: real 0.006s user 0.002s sys 0.004s
W20260925 22:47:18.889535 25795 tablet.cc:2369] T ed68a1581d914821b50a865996d12512 P e6fad9ce2c614cb08498cd3cdf453424: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 22:47:18.897101 25663 tablet.cc:2369] T ed68a1581d914821b50a865996d12512 P e2485ed683004618a230d0a403a4d008: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 22:47:18.930371 25525 tablet.cc:2369] T ed68a1581d914821b50a865996d12512 P d446d20efa5e403c9ca66ad9e6bd6829: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 22:47:18.999571 25811 raft_consensus.cc:482] T ed68a1581d914821b50a865996d12512 P e2485ed683004618a230d0a403a4d008 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 22:47:18.999922 25811 raft_consensus.cc:504] T ed68a1581d914821b50a865996d12512 P e2485ed683004618a230d0a403a4d008 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2485ed683004618a230d0a403a4d008" member_type: VOTER last_known_addr { host: "127.10.198.2" port: 35091 } } peers { permanent_uuid: "e6fad9ce2c614cb08498cd3cdf453424" member_type: VOTER last_known_addr { host: "127.10.198.3" port: 37747 } } peers { permanent_uuid: "d446d20efa5e403c9ca66ad9e6bd6829" member_type: VOTER last_known_addr { host: "127.10.198.1" port: 35361 } }
I20260925 22:47:19.001487 25811 leader_election.cc:288] T ed68a1581d914821b50a865996d12512 P e2485ed683004618a230d0a403a4d008 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e6fad9ce2c614cb08498cd3cdf453424 (127.10.198.3:37747), d446d20efa5e403c9ca66ad9e6bd6829 (127.10.198.1:35361)
I20260925 22:47:19.006381 25813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:47:19.001891 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.198.1:35361 (local address 127.10.198.2:42505)
0925 22:47:19.002333 (+ 442us) negotiation.cc:107] Waiting for socket to connect
0925 22:47:19.002347 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 22:47:19.002493 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:47:19.002832 (+ 339us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:47:19.002835 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:47:19.002885 (+ 50us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:47:19.003032 (+ 147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:47:19.003043 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:47:19.003867 (+ 824us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:47:19.003871 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:47:19.004860 (+ 989us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:47:19.004873 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:47:19.005848 (+ 975us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:47:19.005852 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:47:19.005967 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:47:19.005992 (+ 25us) client_negotiation.cc:769] Sending connection context
0925 22:47:19.006084 (+ 92us) client_negotiation.cc:240] Negotiation successful
0925 22:47:19.006177 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":366,"mutex_wait_us":73,"spinlock_wait_cycles":4736,"thread_start_us":82,"threads_started":1}
I20260925 22:47:19.006356 25815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:47:19.001986 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.198.2:42505 (local address 127.10.198.1:35361)
0925 22:47:19.002294 (+ 308us) server_negotiation.cc:206] Beginning negotiation
0925 22:47:19.002299 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:47:19.002448 (+ 149us) server_negotiation.cc:407] Connection header received
0925 22:47:19.002623 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:47:19.002626 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:47:19.002693 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:47:19.002821 (+ 128us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:47:19.003171 (+ 350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:47:19.003704 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:47:19.005176 (+ 1472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:47:19.005673 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:47:19.005847 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:47:19.006015 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0925 22:47:19.006099 (+ 84us) server_negotiation.cc:299] Negotiation successful
0925 22:47:19.006177 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":229,"thread_start_us":97,"threads_started":1}
I20260925 22:47:19.006414 25805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:47:19.001975 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.198.2:53171 (local address 127.10.198.3:37747)
0925 22:47:19.002150 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0925 22:47:19.002156 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 22:47:19.002394 (+ 238us) server_negotiation.cc:407] Connection header received
0925 22:47:19.002620 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:47:19.002626 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:47:19.002687 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:47:19.002806 (+ 119us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:47:19.003222 (+ 416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:47:19.003803 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:47:19.005142 (+ 1339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:47:19.005645 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:47:19.005853 (+ 208us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:47:19.005942 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0925 22:47:19.006172 (+ 230us) server_negotiation.cc:299] Negotiation successful
0925 22:47:19.006264 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260925 22:47:19.006460 25814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:47:19.001747 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.198.3:37747 (local address 127.10.198.2:53171)
0925 22:47:19.002246 (+ 499us) negotiation.cc:107] Waiting for socket to connect
0925 22:47:19.002265 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0925 22:47:19.002475 (+ 210us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:47:19.002825 (+ 350us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:47:19.002828 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:47:19.002884 (+ 56us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:47:19.003073 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:47:19.003080 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:47:19.004002 (+ 922us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:47:19.004008 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:47:19.004960 (+ 952us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:47:19.004971 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:47:19.005849 (+ 878us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:47:19.005854 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:47:19.006006 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:47:19.006029 (+ 23us) client_negotiation.cc:769] Sending connection context
0925 22:47:19.006137 (+ 108us) client_negotiation.cc:240] Negotiation successful
0925 22:47:19.006231 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":375,"mutex_wait_us":20,"spinlock_wait_cycles":37120,"thread_start_us":114,"threads_started":1}
I20260925 22:47:19.007249 25748 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed68a1581d914821b50a865996d12512" candidate_uuid: "e2485ed683004618a230d0a403a4d008" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e6fad9ce2c614cb08498cd3cdf453424" is_pre_election: true
I20260925 22:47:19.007500 25478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed68a1581d914821b50a865996d12512" candidate_uuid: "e2485ed683004618a230d0a403a4d008" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d446d20efa5e403c9ca66ad9e6bd6829" is_pre_election: true
I20260925 22:47:19.007700 25748 raft_consensus.cc:2454] T ed68a1581d914821b50a865996d12512 P e6fad9ce2c614cb08498cd3cdf453424 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2485ed683004618a230d0a403a4d008 in term 0.
I20260925 22:47:19.007752 25478 raft_consensus.cc:2454] T ed68a1581d914821b50a865996d12512 P d446d20efa5e403c9ca66ad9e6bd6829 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2485ed683004618a230d0a403a4d008 in term 0.
I20260925 22:47:19.008355 25551 leader_election.cc:302] T ed68a1581d914821b50a865996d12512 P e2485ed683004618a230d0a403a4d008 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: d446d20efa5e403c9ca66ad9e6bd6829, e2485ed683004618a230d0a403a4d008, e6fad9ce2c614cb08498cd3cdf453424; no voters:
I20260925 22:47:19.008715 25811 raft_consensus.cc:2781] T ed68a1581d914821b50a865996d12512 P e2485ed683004618a230d0a403a4d008 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 22:47:19.008834 25811 raft_consensus.cc:482] T ed68a1581d914821b50a865996d12512 P e2485ed683004618a230d0a403a4d008 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 22:47:19.008900 25811 raft_consensus.cc:3037] T ed68a1581d914821b50a865996d12512 P e2485ed683004618a230d0a403a4d008 [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:47:19.010032 25811 raft_consensus.cc:504] T ed68a1581d914821b50a865996d12512 P e2485ed683004618a230d0a403a4d008 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2485ed683004618a230d0a403a4d008" member_type: VOTER last_known_addr { host: "127.10.198.2" port: 35091 } } peers { permanent_uuid: "e6fad9ce2c614cb08498cd3cdf453424" member_type: VOTER last_known_addr { host: "127.10.198.3" port: 37747 } } peers { permanent_uuid: "d446d20efa5e403c9ca66ad9e6bd6829" member_type: VOTER last_known_addr { host: "127.10.198.1" port: 35361 } }
I20260925 22:47:19.010465 25811 leader_election.cc:288] T ed68a1581d914821b50a865996d12512 P e2485ed683004618a230d0a403a4d008 [CANDIDATE]: Term 1 election: Requested vote from peers e6fad9ce2c614cb08498cd3cdf453424 (127.10.198.3:37747), d446d20efa5e403c9ca66ad9e6bd6829 (127.10.198.1:35361)
I20260925 22:47:19.010964 25748 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed68a1581d914821b50a865996d12512" candidate_uuid: "e2485ed683004618a230d0a403a4d008" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e6fad9ce2c614cb08498cd3cdf453424"
I20260925 22:47:19.010964 25478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed68a1581d914821b50a865996d12512" candidate_uuid: "e2485ed683004618a230d0a403a4d008" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d446d20efa5e403c9ca66ad9e6bd6829"
I20260925 22:47:19.011124 25478 raft_consensus.cc:3037] T ed68a1581d914821b50a865996d12512 P d446d20efa5e403c9ca66ad9e6bd6829 [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:47:19.011124 25748 raft_consensus.cc:3037] T ed68a1581d914821b50a865996d12512 P e6fad9ce2c614cb08498cd3cdf453424 [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:47:19.012562 25748 raft_consensus.cc:2454] T ed68a1581d914821b50a865996d12512 P e6fad9ce2c614cb08498cd3cdf453424 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2485ed683004618a230d0a403a4d008 in term 1.
I20260925 22:47:19.012562 25478 raft_consensus.cc:2454] T ed68a1581d914821b50a865996d12512 P d446d20efa5e403c9ca66ad9e6bd6829 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2485ed683004618a230d0a403a4d008 in term 1.
I20260925 22:47:19.012995 25550 leader_election.cc:302] T ed68a1581d914821b50a865996d12512 P e2485ed683004618a230d0a403a4d008 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d446d20efa5e403c9ca66ad9e6bd6829, e2485ed683004618a230d0a403a4d008; no voters:
I20260925 22:47:19.013248 25811 raft_consensus.cc:2781] T ed68a1581d914821b50a865996d12512 P e2485ed683004618a230d0a403a4d008 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 22:47:19.013504 25811 raft_consensus.cc:686] T ed68a1581d914821b50a865996d12512 P e2485ed683004618a230d0a403a4d008 [term 1 LEADER]: Becoming Leader. State: Replica: e2485ed683004618a230d0a403a4d008, State: Running, Role: LEADER
I20260925 22:47:19.013875 25811 consensus_queue.cc:237] T ed68a1581d914821b50a865996d12512 P e2485ed683004618a230d0a403a4d008 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2485ed683004618a230d0a403a4d008" member_type: VOTER last_known_addr { host: "127.10.198.2" port: 35091 } } peers { permanent_uuid: "e6fad9ce2c614cb08498cd3cdf453424" member_type: VOTER last_known_addr { host: "127.10.198.3" port: 37747 } } peers { permanent_uuid: "d446d20efa5e403c9ca66ad9e6bd6829" member_type: VOTER last_known_addr { host: "127.10.198.1" port: 35361 } }
I20260925 22:47:19.017395 25335 catalog_manager.cc:5495] T ed68a1581d914821b50a865996d12512 P e2485ed683004618a230d0a403a4d008 reported cstate change: term changed from 0 to 1, leader changed from <none> to e2485ed683004618a230d0a403a4d008 (127.10.198.2). New cstate: current_term: 1 leader_uuid: "e2485ed683004618a230d0a403a4d008" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2485ed683004618a230d0a403a4d008" member_type: VOTER last_known_addr { host: "127.10.198.2" port: 35091 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e6fad9ce2c614cb08498cd3cdf453424" member_type: VOTER last_known_addr { host: "127.10.198.3" port: 37747 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d446d20efa5e403c9ca66ad9e6bd6829" member_type: VOTER last_known_addr { host: "127.10.198.1" port: 35361 } health_report { overall_health: UNKNOWN } } }
I20260925 22:47:19.069459 25803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:47:19.065018 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36950 (local address 127.10.198.2:35091)
0925 22:47:19.065202 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0925 22:47:19.065206 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:47:19.065223 (+ 17us) server_negotiation.cc:407] Connection header received
0925 22:47:19.065360 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:47:19.065363 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:47:19.065436 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:47:19.065572 (+ 136us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 22:47:19.065846 (+ 274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:47:19.066360 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:47:19.067461 (+ 1101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:47:19.067747 (+ 286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:47:19.067916 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:47:19.068418 (+ 502us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 22:47:19.068932 (+ 514us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 22:47:19.069049 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0925 22:47:19.069175 (+ 126us) server_negotiation.cc:299] Negotiation successful
0925 22:47:19.069283 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":95}
I20260925 22:47:19.076287 25478 raft_consensus.cc:1261] T ed68a1581d914821b50a865996d12512 P d446d20efa5e403c9ca66ad9e6bd6829 [term 1 FOLLOWER]: Refusing update from remote peer e2485ed683004618a230d0a403a4d008: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 22:47:19.076288 25748 raft_consensus.cc:1261] T ed68a1581d914821b50a865996d12512 P e6fad9ce2c614cb08498cd3cdf453424 [term 1 FOLLOWER]: Refusing update from remote peer e2485ed683004618a230d0a403a4d008: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 22:47:19.077229 25816 consensus_queue.cc:1035] T ed68a1581d914821b50a865996d12512 P e2485ed683004618a230d0a403a4d008 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d446d20efa5e403c9ca66ad9e6bd6829" member_type: VOTER last_known_addr { host: "127.10.198.1" port: 35361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 22:47:19.077471 25811 consensus_queue.cc:1035] T ed68a1581d914821b50a865996d12512 P e2485ed683004618a230d0a403a4d008 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e6fad9ce2c614cb08498cd3cdf453424" member_type: VOTER last_known_addr { host: "127.10.198.3" port: 37747 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 22:47:19.085325 25820 mvcc.cc:205] Tried to move back new op lower bound from 7333381894447849472 to 7333381894203682816. Current Snapshot: MvccSnapshot[applied={T|T < 7333381894447849472}]
I20260925 22:47:19.094059 25336 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:38626:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 22:47:19.094237 25336 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:38626:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 22:47:19.097239 25336 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 27eb88732a4e4caaae541c787b446b72: Sending DeleteTablet for 3 replicas of tablet ed68a1581d914821b50a865996d12512
I20260925 22:47:19.098212 25456 tablet_service.cc:1514] Processing DeleteTablet for tablet ed68a1581d914821b50a865996d12512 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 22:47:19 UTC) from {username='kudu', principal='kudu/127.10.198.62@KRBTEST.COM'} at 127.0.0.1:39394
I20260925 22:47:19.098349 25596 tablet_service.cc:1514] Processing DeleteTablet for tablet ed68a1581d914821b50a865996d12512 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 22:47:19 UTC) from {username='kudu', principal='kudu/127.10.198.62@KRBTEST.COM'} at 127.0.0.1:36948
I20260925 22:47:19.098492 25728 tablet_service.cc:1514] Processing DeleteTablet for tablet ed68a1581d914821b50a865996d12512 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 22:47:19 UTC) from {username='kudu', principal='kudu/127.10.198.62@KRBTEST.COM'} at 127.0.0.1:42308
I20260925 22:47:19.098798 11032 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrWXm8X/build/debug/bin/kudu with pid 25374
I20260925 22:47:19.098917 25829 tablet_replica.cc:330] T ed68a1581d914821b50a865996d12512 P d446d20efa5e403c9ca66ad9e6bd6829: stopping tablet replica
I20260925 22:47:19.099042 25829 raft_consensus.cc:2229] T ed68a1581d914821b50a865996d12512 P d446d20efa5e403c9ca66ad9e6bd6829 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 22:47:19.099174 25829 raft_consensus.cc:2258] T ed68a1581d914821b50a865996d12512 P d446d20efa5e403c9ca66ad9e6bd6829 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 22:47:19.099705 25830 tablet_replica.cc:330] T ed68a1581d914821b50a865996d12512 P e2485ed683004618a230d0a403a4d008: stopping tablet replica
I20260925 22:47:19.100075 25830 raft_consensus.cc:2229] T ed68a1581d914821b50a865996d12512 P e2485ed683004618a230d0a403a4d008 [term 1 LEADER]: Raft consensus shutting down.
I20260925 22:47:19.100457 25830 raft_consensus.cc:2258] T ed68a1581d914821b50a865996d12512 P e2485ed683004618a230d0a403a4d008 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 22:47:19.102540 25830 ts_tablet_manager.cc:1905] T ed68a1581d914821b50a865996d12512 P e2485ed683004618a230d0a403a4d008: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 22:47:19.102571 25831 tablet_replica.cc:330] T ed68a1581d914821b50a865996d12512 P e6fad9ce2c614cb08498cd3cdf453424: stopping tablet replica
I20260925 22:47:19.103040 25831 raft_consensus.cc:2229] T ed68a1581d914821b50a865996d12512 P e6fad9ce2c614cb08498cd3cdf453424 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 22:47:19.103473 25831 raft_consensus.cc:2258] T ed68a1581d914821b50a865996d12512 P e6fad9ce2c614cb08498cd3cdf453424 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 22:47:19.105729 25831 ts_tablet_manager.cc:1905] T ed68a1581d914821b50a865996d12512 P e6fad9ce2c614cb08498cd3cdf453424: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 22:47:19.106638 25830 ts_tablet_manager.cc:1918] T ed68a1581d914821b50a865996d12512 P e2485ed683004618a230d0a403a4d008: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 22:47:19.106773 25830 log.cc:1196] T ed68a1581d914821b50a865996d12512 P e2485ed683004618a230d0a403a4d008: Deleting WAL directory at /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376259650642-11032-0/minicluster-data/ts-1/wal/wals/ed68a1581d914821b50a865996d12512
I20260925 22:47:19.107198 25830 ts_tablet_manager.cc:1939] T ed68a1581d914821b50a865996d12512 P e2485ed683004618a230d0a403a4d008: Deleting consensus metadata
W20260925 22:47:19.107432 25321 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.10.198.1:35361 (error 108)
I20260925 22:47:19.108495 11032 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrWXm8X/build/debug/bin/kudu with pid 25528
I20260925 22:47:19.110307 25321 catalog_manager.cc:4841] TS e2485ed683004618a230d0a403a4d008 (127.10.198.2:35091): tablet ed68a1581d914821b50a865996d12512 (table test-table [id=2dc15b00d9af44489869d3c0c85c1d0c]) successfully deleted
I20260925 22:47:19.111074 25804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:47:19.110747 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.198.1:35361 (local address 127.0.0.1:39396)
0925 22:47:19.110902 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0925 22:47:19.110984 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.198.1:35361: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
W20260925 22:47:19.111311 25321 catalog_manager.cc:4584] TS d446d20efa5e403c9ca66ad9e6bd6829 (127.10.198.1:35361): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet ed68a1581d914821b50a865996d12512: Network error: Client connection negotiation failed: client connection to 127.10.198.1:35361: connect: Connection refused (error 111)
I20260925 22:47:19.112190 25831 ts_tablet_manager.cc:1918] T ed68a1581d914821b50a865996d12512 P e6fad9ce2c614cb08498cd3cdf453424: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 22:47:19.112366 25831 log.cc:1196] T ed68a1581d914821b50a865996d12512 P e6fad9ce2c614cb08498cd3cdf453424: Deleting WAL directory at /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376259650642-11032-0/minicluster-data/ts-2/wal/wals/ed68a1581d914821b50a865996d12512
I20260925 22:47:19.112807 25831 ts_tablet_manager.cc:1939] T ed68a1581d914821b50a865996d12512 P e6fad9ce2c614cb08498cd3cdf453424: Deleting consensus metadata
I20260925 22:47:19.114701 25322 catalog_manager.cc:4841] TS e6fad9ce2c614cb08498cd3cdf453424 (127.10.198.3:37747): tablet ed68a1581d914821b50a865996d12512 (table test-table [id=2dc15b00d9af44489869d3c0c85c1d0c]) successfully deleted
I20260925 22:47:19.116640 11032 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrWXm8X/build/debug/bin/kudu with pid 25665
I20260925 22:47:19.122985 11032 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrWXm8X/build/debug/bin/kudu with pid 25303
I20260925 22:47:19.130872 11032 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrWXm8X/build/debug/bin/kudu
/tmp/dist-test-taskrWXm8X/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376259650642-11032-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376259650642-11032-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376259650642-11032-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376259650642-11032-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.198.62:42687
--webserver_interface=127.10.198.62
--webserver_port=36809
--keytab_file=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376259650642-11032-0/krb5kdc/kudu_127.10.198.62.keytab
--principal=kudu/127.10.198.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.198.20:44287
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.198.62:42687
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376259650642-11032-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376259650642-11032-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376259650642-11032-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:47:19.255474 25833 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:47:19.255823 25833 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:47:19.256111 25833 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:47:19.260792 25833 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 22:47:19.260882 25833 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:47:19.260903 25833 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:47:19.260921 25833 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 22:47:19.260937 25833 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 22:47:19.267093 25833 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.198.20:44287
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376259650642-11032-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376259650642-11032-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.198.62:42687
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376259650642-11032-0/krb5kdc/kudu_127.10.198.62.keytab
--principal=kudu/127.10.198.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.198.62:42687
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376259650642-11032-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.198.62
--webserver_port=36809
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25833
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376259650642-11032-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:09:16 UTC on ccycloud.quasar-afmhyw.root.comops.site
build id 4673379
I20260925 22:47:19.268481 25833 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:47:19.269697 25833 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:47:19.279263 25841 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:47:19.279255 25838 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:47:19.279554 25833 server_base.cc:1034] running on GCE node
W20260925 22:47:19.279255 25839 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:47:19.280440 25833 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:47:19.281776 25833 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:47:19.283037 25833 hybrid_clock.cc:648] HybridClock initialized: now 1790376439283000 us; error 70 us; skew 500 ppm
Sep 25 22:47:19 dist-test-slave-5vdr krb5kdc[24755](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376439, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.198.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:47:19.286408 25833 init.cc:377] Logged in from keytab as kudu/127.10.198.62@KRBTEST.COM (short username kudu)
I20260925 22:47:19.287539 25833 webserver.cc:460] Webserver started at http://127.10.198.62:36809/ using document root <none> and password file <none>
I20260925 22:47:19.288285 25833 fs_manager.cc:359] Metadata directory not provided
I20260925 22:47:19.288362 25833 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:47:19.292845 25833 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260925 22:47:19.295511 25848 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:47:19.297153 25833 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 22:47:19.297358 25833 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376259650642-11032-0/minicluster-data/master-0/data,/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376259650642-11032-0/minicluster-data/master-0/wal
uuid: "27eb88732a4e4caaae541c787b446b72"
format_stamp: "Formatted at 2026-09-25 22:47:16 on dist-test-slave-5vdr"
server_key: "e87b98d930f8e15835be623fbc0bd542"
server_key_iv: "802d4f5a06d643fdbf4e2c978d2ee152"
server_key_version: "encryptionkey@0"
I20260925 22:47:19.298089 25833 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376259650642-11032-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376259650642-11032-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376259650642-11032-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:47:19.319154 25833 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:47:19.323099 25833 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:47:19.323411 25833 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:47:19.332823 25833 rpc_server.cc:307] RPC server started. Bound to: 127.10.198.62:42687
I20260925 22:47:19.333133 25900 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.198.62:42687 every 8 connection(s)
I20260925 22:47:19.334302 25833 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376259650642-11032-0/minicluster-data/master-0/data/info.pb
I20260925 22:47:19.338694 11032 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrWXm8X/build/debug/bin/kudu as pid 25833
I20260925 22:47:19.339174 11032 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrWXm8X/build/debug/bin/kudu
/tmp/dist-test-taskrWXm8X/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376259650642-11032-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376259650642-11032-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376259650642-11032-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376259650642-11032-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.198.1:35361
--local_ip_for_outbound_sockets=127.10.198.1
--tserver_master_addrs=127.10.198.62:42687
--webserver_port=35377
--webserver_interface=127.10.198.1
--keytab_file=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376259650642-11032-0/krb5kdc/kudu_127.10.198.1.keytab
--principal=kudu/127.10.198.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.198.20:44287
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376259650642-11032-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376259650642-11032-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376259650642-11032-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 22:47:19.339267 25901 sys_catalog.cc:263] Verifying existing consensus state
I20260925 22:47:19.341212 25901 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 27eb88732a4e4caaae541c787b446b72: Bootstrap starting.
I20260925 22:47:19.360314 25901 log.cc:824] T 00000000000000000000000000000000 P 27eb88732a4e4caaae541c787b446b72: Log is configured to *not* fsync() on all Append() calls
I20260925 22:47:19.373944 25901 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 27eb88732a4e4caaae541c787b446b72: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 22:47:19.374449 25901 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 27eb88732a4e4caaae541c787b446b72: Bootstrap complete.
I20260925 22:47:19.379274 25901 raft_consensus.cc:348] T 00000000000000000000000000000000 P 27eb88732a4e4caaae541c787b446b72 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "27eb88732a4e4caaae541c787b446b72" member_type: VOTER last_known_addr { host: "127.10.198.62" port: 42687 } }
I20260925 22:47:19.379830 25901 raft_consensus.cc:729] T 00000000000000000000000000000000 P 27eb88732a4e4caaae541c787b446b72 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 27eb88732a4e4caaae541c787b446b72, State: Initialized, Role: FOLLOWER
I20260925 22:47:19.380502 25901 consensus_queue.cc:260] T 00000000000000000000000000000000 P 27eb88732a4e4caaae541c787b446b72 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "27eb88732a4e4caaae541c787b446b72" member_type: VOTER last_known_addr { host: "127.10.198.62" port: 42687 } }
I20260925 22:47:19.380766 25901 raft_consensus.cc:388] T 00000000000000000000000000000000 P 27eb88732a4e4caaae541c787b446b72 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 22:47:19.380867 25901 raft_consensus.cc:482] T 00000000000000000000000000000000 P 27eb88732a4e4caaae541c787b446b72 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 22:47:19.380996 25901 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 27eb88732a4e4caaae541c787b446b72 [term 2 FOLLOWER]: Advancing to term 3
I20260925 22:47:19.383489 25901 raft_consensus.cc:504] T 00000000000000000000000000000000 P 27eb88732a4e4caaae541c787b446b72 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "27eb88732a4e4caaae541c787b446b72" member_type: VOTER last_known_addr { host: "127.10.198.62" port: 42687 } }
I20260925 22:47:19.383994 25901 leader_election.cc:302] T 00000000000000000000000000000000 P 27eb88732a4e4caaae541c787b446b72 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 27eb88732a4e4caaae541c787b446b72; no voters:
I20260925 22:47:19.384426 25901 leader_election.cc:288] T 00000000000000000000000000000000 P 27eb88732a4e4caaae541c787b446b72 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260925 22:47:19.384596 25905 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 27eb88732a4e4caaae541c787b446b72 [term 3 FOLLOWER]: Leader election won for term 3
I20260925 22:47:19.384881 25905 raft_consensus.cc:686] T 00000000000000000000000000000000 P 27eb88732a4e4caaae541c787b446b72 [term 3 LEADER]: Becoming Leader. State: Replica: 27eb88732a4e4caaae541c787b446b72, State: Running, Role: LEADER
I20260925 22:47:19.385372 25905 consensus_queue.cc:237] T 00000000000000000000000000000000 P 27eb88732a4e4caaae541c787b446b72 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "27eb88732a4e4caaae541c787b446b72" member_type: VOTER last_known_addr { host: "127.10.198.62" port: 42687 } }
I20260925 22:47:19.385974 25901 sys_catalog.cc:564] T 00000000000000000000000000000000 P 27eb88732a4e4caaae541c787b446b72 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 22:47:19.386746 25905 sys_catalog.cc:455] T 00000000000000000000000000000000 P 27eb88732a4e4caaae541c787b446b72 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "27eb88732a4e4caaae541c787b446b72" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "27eb88732a4e4caaae541c787b446b72" member_type: VOTER last_known_addr { host: "127.10.198.62" port: 42687 } } }
I20260925 22:47:19.387051 25905 sys_catalog.cc:458] T 00000000000000000000000000000000 P 27eb88732a4e4caaae541c787b446b72 [sys.catalog]: This master's current role is: LEADER
I20260925 22:47:19.387336 25905 sys_catalog.cc:455] T 00000000000000000000000000000000 P 27eb88732a4e4caaae541c787b446b72 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 27eb88732a4e4caaae541c787b446b72. Latest consensus state: current_term: 3 leader_uuid: "27eb88732a4e4caaae541c787b446b72" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "27eb88732a4e4caaae541c787b446b72" member_type: VOTER last_known_addr { host: "127.10.198.62" port: 42687 } } }
I20260925 22:47:19.387503 25905 sys_catalog.cc:458] T 00000000000000000000000000000000 P 27eb88732a4e4caaae541c787b446b72 [sys.catalog]: This master's current role is: LEADER
I20260925 22:47:19.387917 25914 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 22:47:19.394554 25914 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 22:47:19.395623 25914 catalog_manager.cc:1259] Loaded cluster ID: f4e6616644f449c08dd601a6389da1b3
I20260925 22:47:19.395737 25914 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260925 22:47:19.396803 25914 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7fd0bca919ad google::LogMessage::Fail() at ??:0
@ 0x7fd0bca9488a google::LogMessage::SendToLog() at ??:0
@ 0x7fd0bca914de google::LogMessage::Flush() at ??:0
@ 0x7fd0bca92a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7fd0c73d0fdd _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_ at ??:0
@ 0x7fd0c73d1d2f kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7fd0c73ce579 _ZZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvENKUlvE_clEv at ??:0
@ 0x7fd0c7412640 _ZNSt17_Function_handlerIFvvEZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7fd0bece33d7 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7fd0bece3c83 _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv at ??:0
@ 0x7fd0bece53f2 _ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7fd0becd4512 kudu::Thread::SuperviseThread() at ??:0
@ 0x7fd0c03186db start_thread at ??:0
@ 0x7fd0bb58f61f clone at ??:0
W20260925 22:47:19.493873 25903 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:47:19.494256 25903 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:47:19.494349 25903 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:47:19.498975 25903 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:47:19.499111 25903 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:47:19.499223 25903 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.198.1
I20260925 22:47:19.505371 25903 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.198.20:44287
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376259650642-11032-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376259650642-11032-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376259650642-11032-0/krb5kdc/kudu_127.10.198.1.keytab
--principal=kudu/127.10.198.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.198.1:35361
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376259650642-11032-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.198.1
--webserver_port=35377
--tserver_master_addrs=127.10.198.62:42687
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25903
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.198.1
--log_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376259650642-11032-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:09:16 UTC on ccycloud.quasar-afmhyw.root.comops.site
build id 4673379
I20260925 22:47:19.506713 25903 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:47:19.507965 25903 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:47:19.522199 25943 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:47:19.522589 25945 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:47:19.522819 25942 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:47:19.523196 25903 server_base.cc:1034] running on GCE node
I20260925 22:47:19.523767 25903 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:47:19.524766 25903 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:47:19.526006 25903 hybrid_clock.cc:648] HybridClock initialized: now 1790376439525900 us; error 124 us; skew 500 ppm
Sep 25 22:47:19 dist-test-slave-5vdr krb5kdc[24755](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376439, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.198.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:47:19.530045 25903 init.cc:377] Logged in from keytab as kudu/127.10.198.1@KRBTEST.COM (short username kudu)
I20260925 22:47:19.531275 25903 webserver.cc:460] Webserver started at http://127.10.198.1:35377/ using document root <none> and password file <none>
I20260925 22:47:19.532331 25903 fs_manager.cc:359] Metadata directory not provided
I20260925 22:47:19.532461 25903 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:47:19.537662 25903 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260925 22:47:19.540401 25952 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:47:19.542066 25903 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 22:47:19.542311 25903 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376259650642-11032-0/minicluster-data/ts-0/data,/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376259650642-11032-0/minicluster-data/ts-0/wal
uuid: "d446d20efa5e403c9ca66ad9e6bd6829"
format_stamp: "Formatted at 2026-09-25 22:47:16 on dist-test-slave-5vdr"
server_key: "d2f2d829885eb12f22e8b0b03c4559fa"
server_key_iv: "39ca185db2e434a48e3305f4d90043b9"
server_key_version: "encryptionkey@0"
I20260925 22:47:19.543087 25903 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376259650642-11032-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376259650642-11032-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376259650642-11032-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:47:19.570320 25903 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:47:19.574401 25903 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:47:19.574668 25903 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:47:19.575354 25903 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 22:47:19.576704 25959 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260925 22:47:19.582945 25903 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260925 22:47:19.583089 25903 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.007s user 0.001s sys 0.000s
I20260925 22:47:19.583194 25903 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260925 22:47:19.585235 25903 ts_tablet_manager.cc:610] Registered 1 tablets
I20260925 22:47:19.585336 25903 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.001s sys 0.000s
I20260925 22:47:19.585713 25959 tablet_bootstrap.cc:492] T ed68a1581d914821b50a865996d12512 P d446d20efa5e403c9ca66ad9e6bd6829: Bootstrap starting.
I20260925 22:47:19.598011 25903 rpc_server.cc:307] RPC server started. Bound to: 127.10.198.1:35361
I20260925 22:47:19.598052 26066 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.198.1:35361 every 8 connection(s)
I20260925 22:47:19.599432 25903 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376259650642-11032-0/minicluster-data/ts-0/data/info.pb
I20260925 22:47:19.608592 11032 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrWXm8X/build/debug/bin/kudu as pid 25903
I20260925 22:47:19.608721 25959 log.cc:824] T ed68a1581d914821b50a865996d12512 P d446d20efa5e403c9ca66ad9e6bd6829: Log is configured to *not* fsync() on all Append() calls
I20260925 22:47:19.609174 11032 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrWXm8X/build/debug/bin/kudu
/tmp/dist-test-taskrWXm8X/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376259650642-11032-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376259650642-11032-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376259650642-11032-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376259650642-11032-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.198.2:35091
--local_ip_for_outbound_sockets=127.10.198.2
--tserver_master_addrs=127.10.198.62:42687
--webserver_port=46157
--webserver_interface=127.10.198.2
--keytab_file=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376259650642-11032-0/krb5kdc/kudu_127.10.198.2.keytab
--principal=kudu/127.10.198.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.198.20:44287
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376259650642-11032-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376259650642-11032-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376259650642-11032-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 22:47:19.619580 25959 tablet_bootstrap.cc:492] T ed68a1581d914821b50a865996d12512 P d446d20efa5e403c9ca66ad9e6bd6829: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 22:47:19.620149 25959 tablet_bootstrap.cc:492] T ed68a1581d914821b50a865996d12512 P d446d20efa5e403c9ca66ad9e6bd6829: Bootstrap complete.
I20260925 22:47:19.621702 25959 ts_tablet_manager.cc:1397] T ed68a1581d914821b50a865996d12512 P d446d20efa5e403c9ca66ad9e6bd6829: Time spent bootstrapping tablet: real 0.036s user 0.026s sys 0.008s
I20260925 22:47:19.625545 25959 raft_consensus.cc:348] T ed68a1581d914821b50a865996d12512 P d446d20efa5e403c9ca66ad9e6bd6829 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2485ed683004618a230d0a403a4d008" member_type: VOTER last_known_addr { host: "127.10.198.2" port: 35091 } } peers { permanent_uuid: "e6fad9ce2c614cb08498cd3cdf453424" member_type: VOTER last_known_addr { host: "127.10.198.3" port: 37747 } } peers { permanent_uuid: "d446d20efa5e403c9ca66ad9e6bd6829" member_type: VOTER last_known_addr { host: "127.10.198.1" port: 35361 } }
I20260925 22:47:19.626241 25959 raft_consensus.cc:729] T ed68a1581d914821b50a865996d12512 P d446d20efa5e403c9ca66ad9e6bd6829 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: d446d20efa5e403c9ca66ad9e6bd6829, State: Initialized, Role: FOLLOWER
I20260925 22:47:19.626888 25959 consensus_queue.cc:260] T ed68a1581d914821b50a865996d12512 P d446d20efa5e403c9ca66ad9e6bd6829 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2485ed683004618a230d0a403a4d008" member_type: VOTER last_known_addr { host: "127.10.198.2" port: 35091 } } peers { permanent_uuid: "e6fad9ce2c614cb08498cd3cdf453424" member_type: VOTER last_known_addr { host: "127.10.198.3" port: 37747 } } peers { permanent_uuid: "d446d20efa5e403c9ca66ad9e6bd6829" member_type: VOTER last_known_addr { host: "127.10.198.1" port: 35361 } }
I20260925 22:47:19.628257 25959 ts_tablet_manager.cc:1428] T ed68a1581d914821b50a865996d12512 P d446d20efa5e403c9ca66ad9e6bd6829: Time spent starting tablet: real 0.006s user 0.007s sys 0.001s
W20260925 22:47:19.669776 26069 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:47:19.602461 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.198.62:42687 (local address 127.10.198.1:33051)
0925 22:47:19.603251 (+ 790us) negotiation.cc:107] Waiting for socket to connect
0925 22:47:19.603326 (+ 75us) client_negotiation.cc:174] Beginning negotiation
0925 22:47:19.604767 (+ 1441us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:47:19.668737 (+ 63970us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.198.62:42687: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":467,"thread_start_us":222,"threads_started":1}
W20260925 22:47:19.670586 26067 heartbeater.cc:644] Failed to heartbeat to 127.10.198.62:42687 (0 consecutive failures): Network error: Failed to ping master at 127.10.198.62:42687: Client connection negotiation failed: client connection to 127.10.198.62:42687: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260925 22:47:19.671586 26069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:47:19.671188 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.198.62:42687 (local address 127.10.198.1:50697)
0925 22:47:19.671357 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0925 22:47:19.671474 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.198.62:42687: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
I20260925 22:47:19.672633 26069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:47:19.672329 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.198.62:42687 (local address 127.10.198.1:37615)
0925 22:47:19.672474 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0925 22:47:19.672565 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.198.62:42687: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
W20260925 22:47:19.672956 26067 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260925 22:47:19.737411 26071 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:47:19.737772 26071 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:47:19.737862 26071 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:47:19.742865 26071 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:47:19.743008 26071 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:47:19.743119 26071 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.198.2
I20260925 22:47:19.749547 26071 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.198.20:44287
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376259650642-11032-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376259650642-11032-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376259650642-11032-0/krb5kdc/kudu_127.10.198.2.keytab
--principal=kudu/127.10.198.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.198.2:35091
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376259650642-11032-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.198.2
--webserver_port=46157
--tserver_master_addrs=127.10.198.62:42687
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26071
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.198.2
--log_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376259650642-11032-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:09:16 UTC on ccycloud.quasar-afmhyw.root.comops.site
build id 4673379
I20260925 22:47:19.750859 26071 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:47:19.752089 26071 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:47:19.763430 26077 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:47:19.763433 26080 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:47:19.763433 26078 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:47:19.763790 26071 server_base.cc:1034] running on GCE node
I20260925 22:47:19.764379 26071 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:47:19.765280 26071 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:47:19.766618 26071 hybrid_clock.cc:648] HybridClock initialized: now 1790376439766596 us; error 48 us; skew 500 ppm
Sep 25 22:47:19 dist-test-slave-5vdr krb5kdc[24755](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376439, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.198.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:47:19.770615 26071 init.cc:377] Logged in from keytab as kudu/127.10.198.2@KRBTEST.COM (short username kudu)
I20260925 22:47:19.771922 26071 webserver.cc:460] Webserver started at http://127.10.198.2:46157/ using document root <none> and password file <none>
I20260925 22:47:19.772639 26071 fs_manager.cc:359] Metadata directory not provided
I20260925 22:47:19.772760 26071 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:47:19.777505 26071 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260925 22:47:19.780537 26087 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:47:19.782059 26071 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 22:47:19.782289 26071 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376259650642-11032-0/minicluster-data/ts-1/data,/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376259650642-11032-0/minicluster-data/ts-1/wal
uuid: "e2485ed683004618a230d0a403a4d008"
format_stamp: "Formatted at 2026-09-25 22:47:17 on dist-test-slave-5vdr"
server_key: "796a5bd9100cbda6ca63715963ec177c"
server_key_iv: "710b02d7903df43cd507750f807f85d8"
server_key_version: "encryptionkey@0"
I20260925 22:47:19.782945 26071 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376259650642-11032-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376259650642-11032-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376259650642-11032-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:47:19.801059 26071 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:47:19.804505 26071 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:47:19.804747 26071 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:47:19.805399 26071 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 22:47:19.806348 26071 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:47:19.806427 26071 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:47:19.806502 26071 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:47:19.806555 26071 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 22:47:19.816816 26071 rpc_server.cc:307] RPC server started. Bound to: 127.10.198.2:35091
I20260925 22:47:19.816833 26200 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.198.2:35091 every 8 connection(s)
I20260925 22:47:19.818059 26071 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376259650642-11032-0/minicluster-data/ts-1/data/info.pb
I20260925 22:47:19.822522 26203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:47:19.820513 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.198.62:42687 (local address 127.10.198.2:33643)
0925 22:47:19.821074 (+ 561us) negotiation.cc:107] Waiting for socket to connect
0925 22:47:19.821212 (+ 138us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.198.62:42687: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":329,"thread_start_us":151,"threads_started":1}
W20260925 22:47:19.823195 26201 heartbeater.cc:644] Failed to heartbeat to 127.10.198.62:42687 (0 consecutive failures): Network error: Failed to ping master at 127.10.198.62:42687: Client connection negotiation failed: client connection to 127.10.198.62:42687: connect: Connection refused (error 111)
I20260925 22:47:19.823945 26203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:47:19.823678 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.198.62:42687 (local address 127.10.198.2:57087)
0925 22:47:19.823799 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0925 22:47:19.823881 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.198.62:42687: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260925 22:47:19.824592 26203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:47:19.824392 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.198.62:42687 (local address 127.10.198.2:59633)
0925 22:47:19.824490 (+ 98us) negotiation.cc:107] Waiting for socket to connect
0925 22:47:19.824540 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.198.62:42687: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260925 22:47:19.824815 26201 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260925 22:47:19.827998 11032 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrWXm8X/build/debug/bin/kudu as pid 26071
I20260925 22:47:19.828445 11032 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrWXm8X/build/debug/bin/kudu
/tmp/dist-test-taskrWXm8X/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376259650642-11032-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376259650642-11032-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376259650642-11032-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376259650642-11032-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.198.3:37747
--local_ip_for_outbound_sockets=127.10.198.3
--tserver_master_addrs=127.10.198.62:42687
--webserver_port=42855
--webserver_interface=127.10.198.3
--keytab_file=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376259650642-11032-0/krb5kdc/kudu_127.10.198.3.keytab
--principal=kudu/127.10.198.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.198.20:44287
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376259650642-11032-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376259650642-11032-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376259650642-11032-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:47:19.948786 26204 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:47:19.949144 26204 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:47:19.949231 26204 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:47:19.953651 26204 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:47:19.953766 26204 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:47:19.953871 26204 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.198.3
I20260925 22:47:19.960192 26204 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.198.20:44287
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376259650642-11032-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376259650642-11032-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376259650642-11032-0/krb5kdc/kudu_127.10.198.3.keytab
--principal=kudu/127.10.198.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.198.3:37747
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376259650642-11032-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.198.3
--webserver_port=42855
--tserver_master_addrs=127.10.198.62:42687
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26204
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.198.3
--log_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376259650642-11032-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:09:16 UTC on ccycloud.quasar-afmhyw.root.comops.site
build id 4673379
I20260925 22:47:19.961409 26204 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:47:19.962553 26204 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:47:19.973110 26210 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:47:19.973111 26209 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:47:19.973261 26204 server_base.cc:1034] running on GCE node
W20260925 22:47:19.973462 26212 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:47:19.973945 26204 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:47:19.974707 26204 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:47:19.975955 26204 hybrid_clock.cc:648] HybridClock initialized: now 1790376439975905 us; error 60 us; skew 500 ppm
Sep 25 22:47:19 dist-test-slave-5vdr krb5kdc[24755](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376439, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.198.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:47:19.979137 26204 init.cc:377] Logged in from keytab as kudu/127.10.198.3@KRBTEST.COM (short username kudu)
I20260925 22:47:19.980273 26204 webserver.cc:460] Webserver started at http://127.10.198.3:42855/ using document root <none> and password file <none>
I20260925 22:47:19.980857 26204 fs_manager.cc:359] Metadata directory not provided
I20260925 22:47:19.980924 26204 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:47:19.985260 26204 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.000s sys 0.004s
I20260925 22:47:19.987787 26219 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:47:19.989349 26204 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260925 22:47:19.989483 26204 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376259650642-11032-0/minicluster-data/ts-2/data,/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376259650642-11032-0/minicluster-data/ts-2/wal
uuid: "e6fad9ce2c614cb08498cd3cdf453424"
format_stamp: "Formatted at 2026-09-25 22:47:17 on dist-test-slave-5vdr"
server_key: "1ebdbebba049bc1ce205275c6fad909d"
server_key_iv: "f7e50512d36c1fc08f3d67e35c186e1f"
server_key_version: "encryptionkey@0"
I20260925 22:47:19.990090 26204 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376259650642-11032-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376259650642-11032-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376259650642-11032-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:47:20.001264 26204 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:47:20.005054 26204 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:47:20.005244 26204 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:47:20.005944 26204 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 22:47:20.006858 26204 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:47:20.006910 26204 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:47:20.006954 26204 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:47:20.006970 26204 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 22:47:20.016923 26204 rpc_server.cc:307] RPC server started. Bound to: 127.10.198.3:37747
I20260925 22:47:20.016966 26332 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.198.3:37747 every 8 connection(s)
I20260925 22:47:20.018085 26204 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790376259650642-11032-0/minicluster-data/ts-2/data/info.pb
I20260925 22:47:20.023039 26335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:47:20.020932 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.198.62:42687 (local address 127.10.198.3:57025)
0925 22:47:20.021573 (+ 641us) negotiation.cc:107] Waiting for socket to connect
0925 22:47:20.021723 (+ 150us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.198.62:42687: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":392,"thread_start_us":173,"threads_started":1}
W20260925 22:47:20.023641 26333 heartbeater.cc:644] Failed to heartbeat to 127.10.198.62:42687 (0 consecutive failures): Network error: Failed to ping master at 127.10.198.62:42687: Client connection negotiation failed: client connection to 127.10.198.62:42687: connect: Connection refused (error 111)
I20260925 22:47:20.024446 26335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:47:20.024181 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.198.62:42687 (local address 127.10.198.3:48137)
0925 22:47:20.024316 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0925 22:47:20.024378 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.198.62:42687: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260925 22:47:20.025185 26335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:47:20.024993 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.198.62:42687 (local address 127.10.198.3:46711)
0925 22:47:20.025091 (+ 98us) negotiation.cc:107] Waiting for socket to connect
0925 22:47:20.025135 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.198.62:42687: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260925 22:47:20.025384 26333 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260925 22:47:20.025921 11032 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrWXm8X/build/debug/bin/kudu as pid 26204
I20260925 22:47:20.674237 26337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:47:20.673744 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.198.62:42687 (local address 127.10.198.1:36683)
0925 22:47:20.674033 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0925 22:47:20.674102 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.198.62:42687: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":204,"thread_start_us":81,"threads_started":1}
I20260925 22:47:20.826113 26338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:47:20.825608 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.198.62:42687 (local address 127.10.198.2:60345)
0925 22:47:20.825912 (+ 304us) negotiation.cc:107] Waiting for socket to connect
0925 22:47:20.825986 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.198.62:42687: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":213,"thread_start_us":81,"threads_started":1}
I20260925 22:47:20.996978 26339 raft_consensus.cc:482] T ed68a1581d914821b50a865996d12512 P d446d20efa5e403c9ca66ad9e6bd6829 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 22:47:20.997224 26339 raft_consensus.cc:504] T ed68a1581d914821b50a865996d12512 P d446d20efa5e403c9ca66ad9e6bd6829 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2485ed683004618a230d0a403a4d008" member_type: VOTER last_known_addr { host: "127.10.198.2" port: 35091 } } peers { permanent_uuid: "e6fad9ce2c614cb08498cd3cdf453424" member_type: VOTER last_known_addr { host: "127.10.198.3" port: 37747 } } peers { permanent_uuid: "d446d20efa5e403c9ca66ad9e6bd6829" member_type: VOTER last_known_addr { host: "127.10.198.1" port: 35361 } }
I20260925 22:47:20.998526 26339 leader_election.cc:288] T ed68a1581d914821b50a865996d12512 P d446d20efa5e403c9ca66ad9e6bd6829 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e2485ed683004618a230d0a403a4d008 (127.10.198.2:35091), e6fad9ce2c614cb08498cd3cdf453424 (127.10.198.3:37747)
Sep 25 22:47:21 dist-test-slave-5vdr krb5kdc[24755](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376439, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.198.1@KRBTEST.COM for kudu/127.10.198.3@KRBTEST.COM
Sep 25 22:47:21 dist-test-slave-5vdr krb5kdc[24755](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376439, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.198.1@KRBTEST.COM for kudu/127.10.198.2@KRBTEST.COM
I20260925 22:47:21.016306 26342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:47:20.999079 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.198.1:34787 (local address 127.10.198.3:37747)
0925 22:47:20.999591 (+ 512us) server_negotiation.cc:206] Beginning negotiation
0925 22:47:20.999604 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0925 22:47:20.999642 (+ 38us) server_negotiation.cc:407] Connection header received
0925 22:47:21.000590 (+ 948us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:47:21.000599 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:47:21.001003 (+ 404us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:47:21.001307 (+ 304us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:47:21.003083 (+ 1776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:47:21.004225 (+ 1142us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:47:21.005718 (+ 1493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:47:21.006260 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:47:21.006472 (+ 212us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:47:21.009914 (+ 3442us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:47:21.009923 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:47:21.009948 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:47:21.010002 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:47:21.012977 (+ 2975us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:47:21.013881 (+ 904us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:47:21.013887 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:47:21.013892 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:47:21.014041 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:47:21.014487 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:47:21.014491 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:47:21.014492 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:47:21.014972 (+ 480us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:47:21.015126 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0925 22:47:21.015762 (+ 636us) server_negotiation.cc:299] Negotiation successful
0925 22:47:21.016089 (+ 327us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":389,"thread_start_us":272,"threads_started":1}
I20260925 22:47:21.016325 26340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:47:20.998862 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.198.3:37747 (local address 127.10.198.1:34787)
0925 22:47:20.999290 (+ 428us) negotiation.cc:107] Waiting for socket to connect
0925 22:47:20.999307 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 22:47:20.999416 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:47:21.001577 (+ 2161us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:47:21.001584 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:47:21.002203 (+ 619us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:47:21.002833 (+ 630us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:47:21.002852 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:47:21.004422 (+ 1570us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:47:21.004425 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:47:21.005530 (+ 1105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:47:21.005542 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:47:21.006426 (+ 884us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:47:21.006430 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:47:21.006702 (+ 272us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:47:21.007518 (+ 816us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:47:21.007531 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:47:21.009510 (+ 1979us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:47:21.013198 (+ 3688us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:47:21.013205 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:47:21.013225 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:47:21.013708 (+ 483us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:47:21.014179 (+ 471us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:47:21.014183 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:47:21.014185 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:47:21.014371 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:47:21.015113 (+ 742us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:47:21.015120 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:47:21.015405 (+ 285us) client_negotiation.cc:769] Sending connection context
0925 22:47:21.015622 (+ 217us) client_negotiation.cc:240] Negotiation successful
0925 22:47:21.016082 (+ 460us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":343,"mutex_wait_us":262,"spinlock_wait_cycles":47872,"thread_start_us":89,"threads_started":1}
I20260925 22:47:21.016332 26341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:47:20.998950 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.198.1:42229 (local address 127.10.198.2:35091)
0925 22:47:20.999768 (+ 818us) server_negotiation.cc:206] Beginning negotiation
0925 22:47:20.999793 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0925 22:47:20.999834 (+ 41us) server_negotiation.cc:407] Connection header received
0925 22:47:21.000656 (+ 822us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:47:21.000663 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:47:21.001003 (+ 340us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:47:21.001521 (+ 518us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:47:21.003080 (+ 1559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:47:21.004224 (+ 1144us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:47:21.005708 (+ 1484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:47:21.006262 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:47:21.006472 (+ 210us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:47:21.010138 (+ 3666us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:47:21.010143 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:47:21.010155 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:47:21.010199 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:47:21.013019 (+ 2820us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:47:21.013871 (+ 852us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:47:21.013885 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:47:21.013892 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:47:21.014041 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:47:21.014474 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:47:21.014478 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:47:21.014480 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:47:21.014971 (+ 491us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:47:21.015126 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0925 22:47:21.015757 (+ 631us) server_negotiation.cc:299] Negotiation successful
0925 22:47:21.016108 (+ 351us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":371,"thread_start_us":103,"threads_started":1}
I20260925 22:47:21.016350 26337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:47:20.998784 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.198.2:35091 (local address 127.10.198.1:42229)
0925 22:47:20.999018 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0925 22:47:20.999039 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0925 22:47:20.999318 (+ 279us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:47:21.001613 (+ 2295us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:47:21.001617 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:47:21.002207 (+ 590us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:47:21.002832 (+ 625us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:47:21.002852 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:47:21.004428 (+ 1576us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:47:21.004431 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:47:21.005528 (+ 1097us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:47:21.005542 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:47:21.006421 (+ 879us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:47:21.006425 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:47:21.006706 (+ 281us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:47:21.007475 (+ 769us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:47:21.007505 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:47:21.009909 (+ 2404us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:47:21.013208 (+ 3299us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:47:21.013212 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:47:21.013228 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:47:21.013714 (+ 486us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:47:21.014169 (+ 455us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:47:21.014172 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:47:21.014174 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:47:21.014360 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:47:21.015113 (+ 753us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:47:21.015120 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:47:21.015405 (+ 285us) client_negotiation.cc:769] Sending connection context
0925 22:47:21.015621 (+ 216us) client_negotiation.cc:240] Negotiation successful
0925 22:47:21.016173 (+ 552us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":137,"mutex_wait_us":376,"spinlock_wait_cycles":40192}
W20260925 22:47:21.019362 25955 leader_election.cc:341] T ed68a1581d914821b50a865996d12512 P d446d20efa5e403c9ca66ad9e6bd6829 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer e6fad9ce2c614cb08498cd3cdf453424 (127.10.198.3:37747): Not found: Tablet not found: ed68a1581d914821b50a865996d12512
W20260925 22:47:21.021013 25954 leader_election.cc:341] T ed68a1581d914821b50a865996d12512 P d446d20efa5e403c9ca66ad9e6bd6829 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer e2485ed683004618a230d0a403a4d008 (127.10.198.2:35091): Not found: Tablet not found: ed68a1581d914821b50a865996d12512
I20260925 22:47:21.021186 25954 leader_election.cc:302] T ed68a1581d914821b50a865996d12512 P d446d20efa5e403c9ca66ad9e6bd6829 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d446d20efa5e403c9ca66ad9e6bd6829; no voters: e2485ed683004618a230d0a403a4d008, e6fad9ce2c614cb08498cd3cdf453424
I20260925 22:47:21.021504 26339 raft_consensus.cc:2726] T ed68a1581d914821b50a865996d12512 P d446d20efa5e403c9ca66ad9e6bd6829 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 22:47:21.026482 26343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:47:21.026025 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.198.62:42687 (local address 127.10.198.3:53313)
0925 22:47:21.026302 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0925 22:47:21.026371 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.198.62:42687: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":197,"thread_start_us":86,"threads_started":1}
I20260925 22:47:21.675938 26344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:47:21.675385 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.198.62:42687 (local address 127.10.198.1:59583)
0925 22:47:21.675681 (+ 296us) negotiation.cc:107] Waiting for socket to connect
0925 22:47:21.675759 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.198.62:42687: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":81,"threads_started":1}
I20260925 22:47:21.827656 26345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:47:21.827161 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.198.62:42687 (local address 127.10.198.2:42993)
0925 22:47:21.827469 (+ 308us) negotiation.cc:107] Waiting for socket to connect
0925 22:47:21.827538 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.198.62:42687: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":214,"thread_start_us":85,"threads_started":1}
I20260925 22:47:22.028769 26346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:47:22.027953 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.198.62:42687 (local address 127.10.198.3:41737)
0925 22:47:22.028412 (+ 459us) negotiation.cc:107] Waiting for socket to connect
0925 22:47:22.028616 (+ 204us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.198.62:42687: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":315,"thread_start_us":81,"threads_started":1}
I20260925 22:47:22.677570 26347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:47:22.677009 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.198.62:42687 (local address 127.10.198.1:42693)
0925 22:47:22.677390 (+ 381us) negotiation.cc:107] Waiting for socket to connect
0925 22:47:22.677464 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.198.62:42687: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":276,"thread_start_us":110,"threads_started":1}
I20260925 22:47:22.692965 26348 raft_consensus.cc:482] T ed68a1581d914821b50a865996d12512 P d446d20efa5e403c9ca66ad9e6bd6829 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 22:47:22.693181 26348 raft_consensus.cc:504] T ed68a1581d914821b50a865996d12512 P d446d20efa5e403c9ca66ad9e6bd6829 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2485ed683004618a230d0a403a4d008" member_type: VOTER last_known_addr { host: "127.10.198.2" port: 35091 } } peers { permanent_uuid: "e6fad9ce2c614cb08498cd3cdf453424" member_type: VOTER last_known_addr { host: "127.10.198.3" port: 37747 } } peers { permanent_uuid: "d446d20efa5e403c9ca66ad9e6bd6829" member_type: VOTER last_known_addr { host: "127.10.198.1" port: 35361 } }
I20260925 22:47:22.693709 26348 leader_election.cc:288] T ed68a1581d914821b50a865996d12512 P d446d20efa5e403c9ca66ad9e6bd6829 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e2485ed683004618a230d0a403a4d008 (127.10.198.2:35091), e6fad9ce2c614cb08498cd3cdf453424 (127.10.198.3:37747)
W20260925 22:47:22.694686 25954 leader_election.cc:341] T ed68a1581d914821b50a865996d12512 P d446d20efa5e403c9ca66ad9e6bd6829 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer e2485ed683004618a230d0a403a4d008 (127.10.198.2:35091): Not found: Tablet not found: ed68a1581d914821b50a865996d12512
W20260925 22:47:22.694867 25955 leader_election.cc:341] T ed68a1581d914821b50a865996d12512 P d446d20efa5e403c9ca66ad9e6bd6829 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer e6fad9ce2c614cb08498cd3cdf453424 (127.10.198.3:37747): Not found: Tablet not found: ed68a1581d914821b50a865996d12512
I20260925 22:47:22.694975 25955 leader_election.cc:302] T ed68a1581d914821b50a865996d12512 P d446d20efa5e403c9ca66ad9e6bd6829 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d446d20efa5e403c9ca66ad9e6bd6829; no voters: e2485ed683004618a230d0a403a4d008, e6fad9ce2c614cb08498cd3cdf453424
I20260925 22:47:22.695155 26348 raft_consensus.cc:2726] T ed68a1581d914821b50a865996d12512 P d446d20efa5e403c9ca66ad9e6bd6829 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 22:47:22.829128 26349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:47:22.828607 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.198.62:42687 (local address 127.10.198.2:35299)
0925 22:47:22.828934 (+ 327us) negotiation.cc:107] Waiting for socket to connect
0925 22:47:22.829006 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.198.62:42687: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":237,"thread_start_us":86,"threads_started":1}
I20260925 22:47:23.030525 26350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:47:23.030041 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.198.62:42687 (local address 127.10.198.3:60161)
0925 22:47:23.030337 (+ 296us) negotiation.cc:107] Waiting for socket to connect
0925 22:47:23.030414 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.198.62:42687: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":203,"thread_start_us":87,"threads_started":1}
I20260925 22:47:23.678925 26351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:47:23.678443 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.198.62:42687 (local address 127.10.198.1:34585)
0925 22:47:23.678722 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0925 22:47:23.678804 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.198.62:42687: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":103,"threads_started":1}
I20260925 22:47:23.830575 26352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:47:23.830077 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.198.62:42687 (local address 127.10.198.2:55839)
0925 22:47:23.830400 (+ 323us) negotiation.cc:107] Waiting for socket to connect
0925 22:47:23.830475 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.198.62:42687: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":234,"thread_start_us":100,"threads_started":1}
I20260925 22:47:24.032166 26353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:47:24.031608 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.198.62:42687 (local address 127.10.198.3:42177)
0925 22:47:24.031977 (+ 369us) negotiation.cc:107] Waiting for socket to connect
0925 22:47:24.032063 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.198.62:42687: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":212,"thread_start_us":87,"threads_started":1}
I20260925 22:47:24.680648 26354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:47:24.680162 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.198.62:42687 (local address 127.10.198.1:32993)
0925 22:47:24.680463 (+ 301us) negotiation.cc:107] Waiting for socket to connect
0925 22:47:24.680548 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.198.62:42687: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":221,"thread_start_us":74,"threads_started":1}
I20260925 22:47:24.832218 26355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:47:24.831671 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.198.62:42687 (local address 127.10.198.2:43903)
0925 22:47:24.832029 (+ 358us) negotiation.cc:107] Waiting for socket to connect
0925 22:47:24.832116 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.198.62:42687: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":277,"thread_start_us":85,"threads_started":1}
I20260925 22:47:25.033602 26356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:47:25.033104 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.198.62:42687 (local address 127.10.198.3:39857)
0925 22:47:25.033422 (+ 318us) negotiation.cc:107] Waiting for socket to connect
0925 22:47:25.033492 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.198.62:42687: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":222,"thread_start_us":84,"threads_started":1}
I20260925 22:47:25.581280 26357 raft_consensus.cc:482] T ed68a1581d914821b50a865996d12512 P d446d20efa5e403c9ca66ad9e6bd6829 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 22:47:25.581422 26357 raft_consensus.cc:504] T ed68a1581d914821b50a865996d12512 P d446d20efa5e403c9ca66ad9e6bd6829 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2485ed683004618a230d0a403a4d008" member_type: VOTER last_known_addr { host: "127.10.198.2" port: 35091 } } peers { permanent_uuid: "e6fad9ce2c614cb08498cd3cdf453424" member_type: VOTER last_known_addr { host: "127.10.198.3" port: 37747 } } peers { permanent_uuid: "d446d20efa5e403c9ca66ad9e6bd6829" member_type: VOTER last_known_addr { host: "127.10.198.1" port: 35361 } }
I20260925 22:47:25.581977 26357 leader_election.cc:288] T ed68a1581d914821b50a865996d12512 P d446d20efa5e403c9ca66ad9e6bd6829 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e2485ed683004618a230d0a403a4d008 (127.10.198.2:35091), e6fad9ce2c614cb08498cd3cdf453424 (127.10.198.3:37747)
W20260925 22:47:25.582863 25954 leader_election.cc:341] T ed68a1581d914821b50a865996d12512 P d446d20efa5e403c9ca66ad9e6bd6829 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer e2485ed683004618a230d0a403a4d008 (127.10.198.2:35091): Not found: Tablet not found: ed68a1581d914821b50a865996d12512
W20260925 22:47:25.583034 25955 leader_election.cc:341] T ed68a1581d914821b50a865996d12512 P d446d20efa5e403c9ca66ad9e6bd6829 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer e6fad9ce2c614cb08498cd3cdf453424 (127.10.198.3:37747): Not found: Tablet not found: ed68a1581d914821b50a865996d12512
I20260925 22:47:25.583124 25955 leader_election.cc:302] T ed68a1581d914821b50a865996d12512 P d446d20efa5e403c9ca66ad9e6bd6829 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d446d20efa5e403c9ca66ad9e6bd6829; no voters: e2485ed683004618a230d0a403a4d008, e6fad9ce2c614cb08498cd3cdf453424
I20260925 22:47:25.583302 26357 raft_consensus.cc:2726] T ed68a1581d914821b50a865996d12512 P d446d20efa5e403c9ca66ad9e6bd6829 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 22:47:25.682036 26358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:47:25.681570 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.198.62:42687 (local address 127.10.198.1:43649)
0925 22:47:25.681841 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0925 22:47:25.681908 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.198.62:42687: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":80,"threads_started":1}
I20260925 22:47:25.833881 26359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:47:25.833282 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.198.62:42687 (local address 127.10.198.2:41933)
0925 22:47:25.833697 (+ 415us) negotiation.cc:107] Waiting for socket to connect
0925 22:47:25.833769 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.198.62:42687: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":317,"thread_start_us":112,"threads_started":1}
I20260925 22:47:26.034924 26360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:47:26.034473 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.198.62:42687 (local address 127.10.198.3:49283)
0925 22:47:26.034767 (+ 294us) negotiation.cc:107] Waiting for socket to connect
0925 22:47:26.034826 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.198.62:42687: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":214,"thread_start_us":89,"threads_started":1}
I20260925 22:47:26.683369 26361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:47:26.682872 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.198.62:42687 (local address 127.10.198.1:43859)
0925 22:47:26.683202 (+ 330us) negotiation.cc:107] Waiting for socket to connect
0925 22:47:26.683270 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.198.62:42687: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":232,"thread_start_us":96,"threads_started":1}
I20260925 22:47:26.835534 26362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:47:26.834929 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.198.62:42687 (local address 127.10.198.2:45713)
0925 22:47:26.835325 (+ 396us) negotiation.cc:107] Waiting for socket to connect
0925 22:47:26.835409 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.198.62:42687: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":300,"thread_start_us":91,"threads_started":1}
I20260925 22:47:27.036377 26363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:47:27.035827 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.198.62:42687 (local address 127.10.198.3:46743)
0925 22:47:27.036208 (+ 381us) negotiation.cc:107] Waiting for socket to connect
0925 22:47:27.036278 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.198.62:42687: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":300,"thread_start_us":74,"threads_started":1}
I20260925 22:47:27.107220 26364 raft_consensus.cc:482] T ed68a1581d914821b50a865996d12512 P d446d20efa5e403c9ca66ad9e6bd6829 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 22:47:27.107359 26364 raft_consensus.cc:504] T ed68a1581d914821b50a865996d12512 P d446d20efa5e403c9ca66ad9e6bd6829 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2485ed683004618a230d0a403a4d008" member_type: VOTER last_known_addr { host: "127.10.198.2" port: 35091 } } peers { permanent_uuid: "e6fad9ce2c614cb08498cd3cdf453424" member_type: VOTER last_known_addr { host: "127.10.198.3" port: 37747 } } peers { permanent_uuid: "d446d20efa5e403c9ca66ad9e6bd6829" member_type: VOTER last_known_addr { host: "127.10.198.1" port: 35361 } }
I20260925 22:47:27.107916 26364 leader_election.cc:288] T ed68a1581d914821b50a865996d12512 P d446d20efa5e403c9ca66ad9e6bd6829 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e2485ed683004618a230d0a403a4d008 (127.10.198.2:35091), e6fad9ce2c614cb08498cd3cdf453424 (127.10.198.3:37747)
W20260925 22:47:27.108750 25955 leader_election.cc:341] T ed68a1581d914821b50a865996d12512 P d446d20efa5e403c9ca66ad9e6bd6829 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer e6fad9ce2c614cb08498cd3cdf453424 (127.10.198.3:37747): Not found: Tablet not found: ed68a1581d914821b50a865996d12512
W20260925 22:47:27.108913 25954 leader_election.cc:341] T ed68a1581d914821b50a865996d12512 P d446d20efa5e403c9ca66ad9e6bd6829 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer e2485ed683004618a230d0a403a4d008 (127.10.198.2:35091): Not found: Tablet not found: ed68a1581d914821b50a865996d12512
I20260925 22:47:27.109009 25954 leader_election.cc:302] T ed68a1581d914821b50a865996d12512 P d446d20efa5e403c9ca66ad9e6bd6829 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d446d20efa5e403c9ca66ad9e6bd6829; no voters: e2485ed683004618a230d0a403a4d008, e6fad9ce2c614cb08498cd3cdf453424
I20260925 22:47:27.109220 26364 raft_consensus.cc:2726] T ed68a1581d914821b50a865996d12512 P d446d20efa5e403c9ca66ad9e6bd6829 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 22:47:27.684881 26365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:47:27.684333 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.198.62:42687 (local address 127.10.198.1:37417)
0925 22:47:27.684693 (+ 360us) negotiation.cc:107] Waiting for socket to connect
0925 22:47:27.684768 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.198.62:42687: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":264,"thread_start_us":82,"threads_started":1}
I20260925 22:47:27.837074 26366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:47:27.836625 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.198.62:42687 (local address 127.10.198.2:39699)
0925 22:47:27.836905 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0925 22:47:27.836968 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.198.62:42687: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":106,"threads_started":1}
I20260925 22:47:28.037748 26367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:47:28.037269 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.198.62:42687 (local address 127.10.198.3:41909)
0925 22:47:28.037574 (+ 305us) negotiation.cc:107] Waiting for socket to connect
0925 22:47:28.037647 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.198.62:42687: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":213,"thread_start_us":79,"threads_started":1}
I20260925 22:47:28.686365 26368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:47:28.685885 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.198.62:42687 (local address 127.10.198.1:54803)
0925 22:47:28.686184 (+ 299us) negotiation.cc:107] Waiting for socket to connect
0925 22:47:28.686265 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.198.62:42687: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":215,"thread_start_us":86,"threads_started":1}
I20260925 22:47:28.838330 26369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:47:28.837907 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.198.62:42687 (local address 127.10.198.2:33671)
0925 22:47:28.838167 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0925 22:47:28.838231 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.198.62:42687: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":80,"threads_started":1}
I20260925 22:47:29.039109 26370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:47:29.038588 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.198.62:42687 (local address 127.10.198.3:56175)
0925 22:47:29.038925 (+ 337us) negotiation.cc:107] Waiting for socket to connect
0925 22:47:29.038994 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.198.62:42687: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":256,"thread_start_us":108,"threads_started":1}
I20260925 22:47:29.688023 26371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:47:29.687433 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.198.62:42687 (local address 127.10.198.1:41755)
0925 22:47:29.687774 (+ 341us) negotiation.cc:107] Waiting for socket to connect
0925 22:47:29.687914 (+ 140us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.198.62:42687: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":251,"thread_start_us":97,"threads_started":1}
I20260925 22:47:29.689507 26372 raft_consensus.cc:482] T ed68a1581d914821b50a865996d12512 P d446d20efa5e403c9ca66ad9e6bd6829 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 22:47:29.689630 26372 raft_consensus.cc:504] T ed68a1581d914821b50a865996d12512 P d446d20efa5e403c9ca66ad9e6bd6829 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2485ed683004618a230d0a403a4d008" member_type: VOTER last_known_addr { host: "127.10.198.2" port: 35091 } } peers { permanent_uuid: "e6fad9ce2c614cb08498cd3cdf453424" member_type: VOTER last_known_addr { host: "127.10.198.3" port: 37747 } } peers { permanent_uuid: "d446d20efa5e403c9ca66ad9e6bd6829" member_type: VOTER last_known_addr { host: "127.10.198.1" port: 35361 } }
I20260925 22:47:29.690086 26372 leader_election.cc:288] T ed68a1581d914821b50a865996d12512 P d446d20efa5e403c9ca66ad9e6bd6829 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e2485ed683004618a230d0a403a4d008 (127.10.198.2:35091), e6fad9ce2c614cb08498cd3cdf453424 (127.10.198.3:37747)
W20260925 22:47:29.690944 25954 leader_election.cc:341] T ed68a1581d914821b50a865996d12512 P d446d20efa5e403c9ca66ad9e6bd6829 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer e2485ed683004618a230d0a403a4d008 (127.10.198.2:35091): Not found: Tablet not found: ed68a1581d914821b50a865996d12512
W20260925 22:47:29.691092 25955 leader_election.cc:341] T ed68a1581d914821b50a865996d12512 P d446d20efa5e403c9ca66ad9e6bd6829 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer e6fad9ce2c614cb08498cd3cdf453424 (127.10.198.3:37747): Not found: Tablet not found: ed68a1581d914821b50a865996d12512
I20260925 22:47:29.691200 25955 leader_election.cc:302] T ed68a1581d914821b50a865996d12512 P d446d20efa5e403c9ca66ad9e6bd6829 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d446d20efa5e403c9ca66ad9e6bd6829; no voters: e2485ed683004618a230d0a403a4d008, e6fad9ce2c614cb08498cd3cdf453424
I20260925 22:47:29.691421 26372 raft_consensus.cc:2726] T ed68a1581d914821b50a865996d12512 P d446d20efa5e403c9ca66ad9e6bd6829 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 22:47:29.839821 26373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:47:29.839322 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.198.62:42687 (local address 127.10.198.2:46997)
0925 22:47:29.839639 (+ 317us) negotiation.cc:107] Waiting for socket to connect
0925 22:47:29.839722 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.198.62:42687: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":226,"thread_start_us":81,"threads_started":1}
I20260925 22:47:30.040746 26374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:47:30.040230 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.198.62:42687 (local address 127.10.198.3:55903)
0925 22:47:30.040574 (+ 344us) negotiation.cc:107] Waiting for socket to connect
0925 22:47:30.040641 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.198.62:42687: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":258,"thread_start_us":80,"threads_started":1}
I20260925 22:47:30.689420 26375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:47:30.688916 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.198.62:42687 (local address 127.10.198.1:52925)
0925 22:47:30.689234 (+ 318us) negotiation.cc:107] Waiting for socket to connect
0925 22:47:30.689309 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.198.62:42687: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":202,"thread_start_us":77,"threads_started":1}
I20260925 22:47:30.841262 26376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:47:30.840790 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.198.62:42687 (local address 127.10.198.2:52695)
0925 22:47:30.841087 (+ 297us) negotiation.cc:107] Waiting for socket to connect
0925 22:47:30.841160 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.198.62:42687: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":204,"thread_start_us":80,"threads_started":1}
I20260925 22:47:31.042162 26377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:47:31.041656 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.198.62:42687 (local address 127.10.198.3:36459)
0925 22:47:31.041988 (+ 332us) negotiation.cc:107] Waiting for socket to connect
0925 22:47:31.042058 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.198.62:42687: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":237,"thread_start_us":104,"threads_started":1}
I20260925 22:47:31.690945 26378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:47:31.690430 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.198.62:42687 (local address 127.10.198.1:36003)
0925 22:47:31.690729 (+ 299us) negotiation.cc:107] Waiting for socket to connect
0925 22:47:31.690799 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.198.62:42687: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":214,"thread_start_us":80,"threads_started":1}
I20260925 22:47:31.842804 26379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:47:31.842336 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.198.62:42687 (local address 127.10.198.2:39611)
0925 22:47:31.842635 (+ 299us) negotiation.cc:107] Waiting for socket to connect
0925 22:47:31.842701 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.198.62:42687: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":215,"thread_start_us":91,"threads_started":1}
I20260925 22:47:32.043659 26380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:47:32.043138 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.198.62:42687 (local address 127.10.198.3:56557)
0925 22:47:32.043487 (+ 349us) negotiation.cc:107] Waiting for socket to connect
0925 22:47:32.043556 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.198.62:42687: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":262,"thread_start_us":97,"threads_started":1}
I20260925 22:47:32.692555 26381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:47:32.692042 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.198.62:42687 (local address 127.10.198.1:51551)
0925 22:47:32.692364 (+ 322us) negotiation.cc:107] Waiting for socket to connect
0925 22:47:32.692437 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.198.62:42687: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":221,"thread_start_us":99,"threads_started":1}
I20260925 22:47:32.844305 26382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:47:32.843830 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.198.62:42687 (local address 127.10.198.2:55211)
0925 22:47:32.844140 (+ 310us) negotiation.cc:107] Waiting for socket to connect
0925 22:47:32.844207 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.198.62:42687: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":78,"threads_started":1}
I20260925 22:47:33.045217 26383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:47:33.044734 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.198.62:42687 (local address 127.10.198.3:34487)
0925 22:47:33.045036 (+ 302us) negotiation.cc:107] Waiting for socket to connect
0925 22:47:33.045107 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.198.62:42687: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":212,"thread_start_us":106,"threads_started":1}
I20260925 22:47:33.694236 26384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:47:33.693539 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.198.62:42687 (local address 127.10.198.1:37805)
0925 22:47:33.693950 (+ 411us) negotiation.cc:107] Waiting for socket to connect
0925 22:47:33.694031 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.198.62:42687: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":314,"thread_start_us":179,"threads_started":1}
I20260925 22:47:33.845778 26385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:47:33.845257 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.198.62:42687 (local address 127.10.198.2:38027)
0925 22:47:33.845604 (+ 347us) negotiation.cc:107] Waiting for socket to connect
0925 22:47:33.845677 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.198.62:42687: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":242,"thread_start_us":121,"threads_started":1}
I20260925 22:47:34.046680 26386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:47:34.046177 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.198.62:42687 (local address 127.10.198.3:34485)
0925 22:47:34.046504 (+ 327us) negotiation.cc:107] Waiting for socket to connect
0925 22:47:34.046577 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.198.62:42687: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":226,"thread_start_us":84,"threads_started":1}
I20260925 22:47:34.695894 26387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:47:34.695373 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.198.62:42687 (local address 127.10.198.1:56219)
0925 22:47:34.695657 (+ 284us) negotiation.cc:107] Waiting for socket to connect
0925 22:47:34.695730 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.198.62:42687: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":199,"thread_start_us":82,"threads_started":1}
I20260925 22:47:34.847347 26388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:47:34.846786 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.198.62:42687 (local address 127.10.198.2:34013)
0925 22:47:34.847102 (+ 316us) negotiation.cc:107] Waiting for socket to connect
0925 22:47:34.847246 (+ 144us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.198.62:42687: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":228,"thread_start_us":79,"threads_started":1}
I20260925 22:47:35.033342 11032 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrWXm8X/build/debug/bin/kudu with pid 25903
I20260925 22:47:35.039580 11032 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrWXm8X/build/debug/bin/kudu with pid 26071
I20260925 22:47:35.045353 11032 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrWXm8X/build/debug/bin/kudu with pid 26204
2026-09-25T22:47:35Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (19882 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376259650642-11032-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 22:47:35 dist-test-slave-5vdr krb5kdc[26391](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 22:47:35 dist-test-slave-5vdr krb5kdc[26391](info): set up 2 sockets
Sep 25 22:47:35 dist-test-slave-5vdr krb5kdc[26391](info): commencing operation
krb5kdc: starting...
W20260925 22:47:36.092248 11032 mini_kdc.cc:121] Time spent starting KDC: real 1.023s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376259650642-11032-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376259650642-11032-0/krb5kdc/test-user.keytab.
Sep 25 22:47:36 dist-test-slave-5vdr krb5kdc[26391](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376456, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T22:47:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T22:47:36Z Disabled control of system clock
WARNING: no policy specified for kudu/127.10.198.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.198.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.198.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376259650642-11032-0/krb5kdc/kudu_127.10.198.62.keytab.
Entry for principal kudu/127.10.198.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376259650642-11032-0/krb5kdc/kudu_127.10.198.62.keytab.
I20260925 22:47:36.228724 11032 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrWXm8X/build/debug/bin/kudu
/tmp/dist-test-taskrWXm8X/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376259650642-11032-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376259650642-11032-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376259650642-11032-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376259650642-11032-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.198.62:41557
--webserver_interface=127.10.198.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376259650642-11032-0/krb5kdc/kudu_127.10.198.62.keytab
--principal=kudu/127.10.198.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.198.20:41079
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.198.62:41557
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376259650642-11032-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376259650642-11032-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376259650642-11032-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:47:36.350515 26404 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:47:36.350854 26404 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:47:36.350901 26404 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:47:36.355412 26404 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260925 22:47:36.355507 26404 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260925 22:47:36.355526 26404 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260925 22:47:36.355541 26404 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 22:47:36.355571 26404 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:47:36.355590 26404 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:47:36.355664 26404 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 22:47:36.355687 26404 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 22:47:36.361757 26404 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.198.20:41079
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376259650642-11032-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376259650642-11032-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.10.198.62:41557
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376259650642-11032-0/krb5kdc/kudu_127.10.198.62.keytab
--principal=kudu/127.10.198.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.198.62:41557
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376259650642-11032-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.198.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26404
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376259650642-11032-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:09:16 UTC on ccycloud.quasar-afmhyw.root.comops.site
build id 4673379
I20260925 22:47:36.363030 26404 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:47:36.364261 26404 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:47:36.373986 26409 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:47:36.374068 26410 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:47:36.374132 26404 server_base.cc:1034] running on GCE node
W20260925 22:47:36.373987 26412 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:47:36.375023 26404 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:47:36.376406 26404 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:47:36.377632 26404 hybrid_clock.cc:648] HybridClock initialized: now 1790376456377601 us; error 56 us; skew 500 ppm
Sep 25 22:47:36 dist-test-slave-5vdr krb5kdc[26391](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376456, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.198.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:47:36.381062 26404 init.cc:377] Logged in from keytab as kudu/127.10.198.62@KRBTEST.COM (short username kudu)
I20260925 22:47:36.382159 26404 webserver.cc:460] Webserver started at http://127.10.198.62:35045/ using document root <none> and password file <none>
I20260925 22:47:36.382797 26404 fs_manager.cc:359] Metadata directory not provided
I20260925 22:47:36.382902 26404 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:47:36.383162 26404 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:47:36.385586 26404 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376259650642-11032-0/minicluster-data/master-0/data/instance:
uuid: "3f1861b6350a4355939000f5afa487a5"
format_stamp: "Formatted at 2026-09-25 22:47:36 on dist-test-slave-5vdr"
server_key: "3e784eed9d2d48329bac4e54677457fc"
server_key_iv: "5da66deeb4410b5f5519f8ab958f454f"
server_key_version: "encryptionkey@0"
I20260925 22:47:36.386288 26404 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376259650642-11032-0/minicluster-data/master-0/wal/instance:
uuid: "3f1861b6350a4355939000f5afa487a5"
format_stamp: "Formatted at 2026-09-25 22:47:36 on dist-test-slave-5vdr"
server_key: "3e784eed9d2d48329bac4e54677457fc"
server_key_iv: "5da66deeb4410b5f5519f8ab958f454f"
server_key_version: "encryptionkey@0"
I20260925 22:47:36.390596 26404 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260925 22:47:36.393630 26419 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:47:36.395021 26404 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260925 22:47:36.395216 26404 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376259650642-11032-0/minicluster-data/master-0/data,/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376259650642-11032-0/minicluster-data/master-0/wal
uuid: "3f1861b6350a4355939000f5afa487a5"
format_stamp: "Formatted at 2026-09-25 22:47:36 on dist-test-slave-5vdr"
server_key: "3e784eed9d2d48329bac4e54677457fc"
server_key_iv: "5da66deeb4410b5f5519f8ab958f454f"
server_key_version: "encryptionkey@0"
I20260925 22:47:36.395370 26404 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376259650642-11032-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376259650642-11032-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376259650642-11032-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:47:36.418128 26404 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:47:36.422260 26404 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:47:36.422544 26404 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:47:36.430500 26404 rpc_server.cc:307] RPC server started. Bound to: 127.10.198.62:41557
I20260925 22:47:36.430887 26471 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.198.62:41557 every 8 connection(s)
I20260925 22:47:36.431622 26404 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376259650642-11032-0/minicluster-data/master-0/data/info.pb
I20260925 22:47:36.434774 26472 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:47:36.437172 11032 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrWXm8X/build/debug/bin/kudu as pid 26404
I20260925 22:47:36.437351 11032 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376259650642-11032-0/minicluster-data/master-0/wal/instance
I20260925 22:47:36.437633 11032 external_mini_cluster.cc:1428] Setting key 145264c7b7076218b186647e4d5e7dd6
I20260925 22:47:36.441949 26472 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3f1861b6350a4355939000f5afa487a5: Bootstrap starting.
I20260925 22:47:36.444916 26472 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3f1861b6350a4355939000f5afa487a5: Neither blocks nor log segments found. Creating new log.
I20260925 22:47:36.445951 26472 log.cc:824] T 00000000000000000000000000000000 P 3f1861b6350a4355939000f5afa487a5: Log is configured to *not* fsync() on all Append() calls
Sep 25 22:47:36 dist-test-slave-5vdr krb5kdc[26391](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376456, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.198.62@KRBTEST.COM
I20260925 22:47:36.448340 26472 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3f1861b6350a4355939000f5afa487a5: No bootstrap required, opened a new log
I20260925 22:47:36.452483 26472 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3f1861b6350a4355939000f5afa487a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f1861b6350a4355939000f5afa487a5" member_type: VOTER last_known_addr { host: "127.10.198.62" port: 41557 } }
I20260925 22:47:36.452798 26472 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3f1861b6350a4355939000f5afa487a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:47:36.452880 26472 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3f1861b6350a4355939000f5afa487a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3f1861b6350a4355939000f5afa487a5, State: Initialized, Role: FOLLOWER
I20260925 22:47:36.453456 26472 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3f1861b6350a4355939000f5afa487a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f1861b6350a4355939000f5afa487a5" member_type: VOTER last_known_addr { host: "127.10.198.62" port: 41557 } }
I20260925 22:47:36.453649 26472 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3f1861b6350a4355939000f5afa487a5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 22:47:36.453743 26472 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3f1861b6350a4355939000f5afa487a5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 22:47:36.453869 26472 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3f1861b6350a4355939000f5afa487a5 [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:47:36.454267 26475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:47:36.439185 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53006 (local address 127.10.198.62:41557)
0925 22:47:36.439686 (+ 501us) server_negotiation.cc:206] Beginning negotiation
0925 22:47:36.439709 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0925 22:47:36.439757 (+ 48us) server_negotiation.cc:407] Connection header received
0925 22:47:36.440677 (+ 920us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:47:36.440703 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:47:36.441160 (+ 457us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:47:36.441571 (+ 411us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:47:36.442413 (+ 842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:47:36.444066 (+ 1653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:47:36.445051 (+ 985us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:47:36.445807 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:47:36.445978 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:47:36.448792 (+ 2814us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:47:36.448798 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:47:36.448814 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:47:36.448845 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:47:36.450774 (+ 1929us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:47:36.451346 (+ 572us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:47:36.451356 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:47:36.451366 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:47:36.451499 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:47:36.451811 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:47:36.451815 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:47:36.451817 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:47:36.452363 (+ 546us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:47:36.452614 (+ 251us) server_negotiation.cc:1035] Waiting for connection context
0925 22:47:36.452901 (+ 287us) server_negotiation.cc:299] Negotiation successful
0925 22:47:36.453290 (+ 389us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":324,"thread_start_us":136,"threads_started":1}
I20260925 22:47:36.455004 26472 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3f1861b6350a4355939000f5afa487a5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f1861b6350a4355939000f5afa487a5" member_type: VOTER last_known_addr { host: "127.10.198.62" port: 41557 } }
I20260925 22:47:36.455380 26472 leader_election.cc:302] T 00000000000000000000000000000000 P 3f1861b6350a4355939000f5afa487a5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3f1861b6350a4355939000f5afa487a5; no voters:
I20260925 22:47:36.455741 26472 leader_election.cc:288] T 00000000000000000000000000000000 P 3f1861b6350a4355939000f5afa487a5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 22:47:36.455916 26477 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3f1861b6350a4355939000f5afa487a5 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 22:47:36.456127 26477 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3f1861b6350a4355939000f5afa487a5 [term 1 LEADER]: Becoming Leader. State: Replica: 3f1861b6350a4355939000f5afa487a5, State: Running, Role: LEADER
I20260925 22:47:36.456677 26477 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3f1861b6350a4355939000f5afa487a5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f1861b6350a4355939000f5afa487a5" member_type: VOTER last_known_addr { host: "127.10.198.62" port: 41557 } }
I20260925 22:47:36.457159 26472 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3f1861b6350a4355939000f5afa487a5 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 22:47:36.458550 26479 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3f1861b6350a4355939000f5afa487a5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3f1861b6350a4355939000f5afa487a5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f1861b6350a4355939000f5afa487a5" member_type: VOTER last_known_addr { host: "127.10.198.62" port: 41557 } } }
I20260925 22:47:36.458585 26478 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3f1861b6350a4355939000f5afa487a5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3f1861b6350a4355939000f5afa487a5. Latest consensus state: current_term: 1 leader_uuid: "3f1861b6350a4355939000f5afa487a5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f1861b6350a4355939000f5afa487a5" member_type: VOTER last_known_addr { host: "127.10.198.62" port: 41557 } } }
I20260925 22:47:36.458715 26478 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3f1861b6350a4355939000f5afa487a5 [sys.catalog]: This master's current role is: LEADER
I20260925 22:47:36.458714 26479 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3f1861b6350a4355939000f5afa487a5 [sys.catalog]: This master's current role is: LEADER
I20260925 22:47:36.459204 26487 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 22:47:36.462368 26487 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 22:47:36.469197 26487 catalog_manager.cc:1347] Generated new cluster ID: ff32c441e4cc4360a3fe479cb6485ef5
I20260925 22:47:36.469292 26487 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 22:47:36.476285 26487 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 22:47:36.477397 26487 catalog_manager.cc:1504] Loading token signing keys...
I20260925 22:47:36.483289 26487 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3f1861b6350a4355939000f5afa487a5: Generated new TSK 0
I20260925 22:47:36.484295 26487 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.10.198.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.198.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.198.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376259650642-11032-0/krb5kdc/kudu_127.10.198.1.keytab.
Entry for principal kudu/127.10.198.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376259650642-11032-0/krb5kdc/kudu_127.10.198.1.keytab.
I20260925 22:47:36.520196 11032 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrWXm8X/build/debug/bin/kudu
/tmp/dist-test-taskrWXm8X/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376259650642-11032-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376259650642-11032-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376259650642-11032-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376259650642-11032-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.198.1:0
--local_ip_for_outbound_sockets=127.10.198.1
--webserver_interface=127.10.198.1
--webserver_port=0
--tserver_master_addrs=127.10.198.62:41557
--keytab_file=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376259650642-11032-0/krb5kdc/kudu_127.10.198.1.keytab
--principal=kudu/127.10.198.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.198.20:41079
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376259650642-11032-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376259650642-11032-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376259650642-11032-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:47:36.646545 26498 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:47:36.646938 26498 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:47:36.647006 26498 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:47:36.651270 26498 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:47:36.651357 26498 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:47:36.651419 26498 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.198.1
I20260925 22:47:36.657299 26498 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.198.20:41079
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376259650642-11032-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376259650642-11032-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376259650642-11032-0/krb5kdc/kudu_127.10.198.1.keytab
--principal=kudu/127.10.198.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.198.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376259650642-11032-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.198.1
--webserver_port=0
--tserver_master_addrs=127.10.198.62:41557
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26498
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.198.1
--log_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376259650642-11032-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:09:16 UTC on ccycloud.quasar-afmhyw.root.comops.site
build id 4673379
I20260925 22:47:36.658614 26498 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:47:36.659744 26498 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:47:36.670475 26503 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:47:36.670475 26504 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:47:36.670503 26506 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:47:36.671072 26498 server_base.cc:1034] running on GCE node
I20260925 22:47:36.671576 26498 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:47:36.672425 26498 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:47:36.673607 26498 hybrid_clock.cc:648] HybridClock initialized: now 1790376456673577 us; error 61 us; skew 500 ppm
Sep 25 22:47:36 dist-test-slave-5vdr krb5kdc[26391](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376456, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.198.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:47:36.677008 26498 init.cc:377] Logged in from keytab as kudu/127.10.198.1@KRBTEST.COM (short username kudu)
I20260925 22:47:36.678112 26498 webserver.cc:460] Webserver started at http://127.10.198.1:34609/ using document root <none> and password file <none>
I20260925 22:47:36.678771 26498 fs_manager.cc:359] Metadata directory not provided
I20260925 22:47:36.678887 26498 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:47:36.679136 26498 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:47:36.681456 26498 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376259650642-11032-0/minicluster-data/ts-0/data/instance:
uuid: "84c93dfd42c94d188c85a636289a4e23"
format_stamp: "Formatted at 2026-09-25 22:47:36 on dist-test-slave-5vdr"
server_key: "e0c3fa3a013ecdfc35c9ebe8f8f04413"
server_key_iv: "c47103792e3bc31abe6ea5c9c990e8e2"
server_key_version: "encryptionkey@0"
I20260925 22:47:36.682121 26498 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376259650642-11032-0/minicluster-data/ts-0/wal/instance:
uuid: "84c93dfd42c94d188c85a636289a4e23"
format_stamp: "Formatted at 2026-09-25 22:47:36 on dist-test-slave-5vdr"
server_key: "e0c3fa3a013ecdfc35c9ebe8f8f04413"
server_key_iv: "c47103792e3bc31abe6ea5c9c990e8e2"
server_key_version: "encryptionkey@0"
I20260925 22:47:36.686534 26498 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260925 22:47:36.689615 26513 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:47:36.691092 26498 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260925 22:47:36.691282 26498 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376259650642-11032-0/minicluster-data/ts-0/data,/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376259650642-11032-0/minicluster-data/ts-0/wal
uuid: "84c93dfd42c94d188c85a636289a4e23"
format_stamp: "Formatted at 2026-09-25 22:47:36 on dist-test-slave-5vdr"
server_key: "e0c3fa3a013ecdfc35c9ebe8f8f04413"
server_key_iv: "c47103792e3bc31abe6ea5c9c990e8e2"
server_key_version: "encryptionkey@0"
I20260925 22:47:36.691435 26498 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376259650642-11032-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376259650642-11032-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376259650642-11032-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:47:36.713557 26498 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:47:36.717145 26498 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:47:36.717391 26498 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:47:36.718061 26498 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 22:47:36.718994 26498 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:47:36.719074 26498 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:47:36.719142 26498 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:47:36.719198 26498 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 22:47:36.728935 26498 rpc_server.cc:307] RPC server started. Bound to: 127.10.198.1:36769
I20260925 22:47:36.728986 26626 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.198.1:36769 every 8 connection(s)
I20260925 22:47:36.730050 26498 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376259650642-11032-0/minicluster-data/ts-0/data/info.pb
I20260925 22:47:36.738659 11032 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrWXm8X/build/debug/bin/kudu as pid 26498
I20260925 22:47:36.738816 11032 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376259650642-11032-0/minicluster-data/ts-0/wal/instance
I20260925 22:47:36.739146 11032 external_mini_cluster.cc:1428] Setting key cae9d0102b14e7d61fe3c1c2d2da6e39
Sep 25 22:47:36 dist-test-slave-5vdr krb5kdc[26391](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376456, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.198.1@KRBTEST.COM for kudu/127.10.198.62@KRBTEST.COM
I20260925 22:47:36.746539 26475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:47:36.732323 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.198.1:47949 (local address 127.10.198.62:41557)
0925 22:47:36.732476 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0925 22:47:36.732479 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 22:47:36.733441 (+ 962us) server_negotiation.cc:407] Connection header received
0925 22:47:36.734784 (+ 1343us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:47:36.734787 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:47:36.734842 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:47:36.734925 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:47:36.736387 (+ 1462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:47:36.736954 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:47:36.738218 (+ 1264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:47:36.738470 (+ 252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:47:36.738577 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:47:36.741737 (+ 3160us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:47:36.741741 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:47:36.741744 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:47:36.741780 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:47:36.743704 (+ 1924us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:47:36.744523 (+ 819us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:47:36.744526 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:47:36.744528 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:47:36.744615 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:47:36.745110 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:47:36.745113 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:47:36.745114 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:47:36.745328 (+ 214us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:47:36.745415 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0925 22:47:36.746257 (+ 842us) server_negotiation.cc:299] Negotiation successful
0925 22:47:36.746390 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260925 22:47:36.747982 26629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:47:36.732745 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.198.62:41557 (local address 127.10.198.1:47949)
0925 22:47:36.733249 (+ 504us) negotiation.cc:107] Waiting for socket to connect
0925 22:47:36.733300 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0925 22:47:36.734537 (+ 1237us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:47:36.735127 (+ 590us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:47:36.735135 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:47:36.735560 (+ 425us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:47:36.736204 (+ 644us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:47:36.736222 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:47:36.737086 (+ 864us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:47:36.737089 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:47:36.738059 (+ 970us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:47:36.738070 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:47:36.738680 (+ 610us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:47:36.738688 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:47:36.738929 (+ 241us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:47:36.739806 (+ 877us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:47:36.739831 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:47:36.741537 (+ 1706us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:47:36.743923 (+ 2386us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:47:36.743934 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:47:36.743958 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:47:36.744402 (+ 444us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:47:36.744725 (+ 323us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:47:36.744731 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:47:36.744736 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:47:36.744957 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:47:36.745459 (+ 502us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:47:36.745471 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:47:36.745909 (+ 438us) client_negotiation.cc:769] Sending connection context
0925 22:47:36.746153 (+ 244us) client_negotiation.cc:240] Negotiation successful
0925 22:47:36.746605 (+ 452us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":303,"thread_start_us":131,"threads_started":1}
I20260925 22:47:36.749641 26627 heartbeater.cc:347] Connected to a master server at 127.10.198.62:41557
I20260925 22:47:36.749934 26627 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.10.198.2@KRBTEST.COM; defaulting to no policy
I20260925 22:47:36.750553 26627 heartbeater.cc:511] Master 127.10.198.62:41557 requested a full tablet report, sending...
I20260925 22:47:36.752411 26436 ts_manager.cc:194] Registered new tserver with Master: 84c93dfd42c94d188c85a636289a4e23 (127.10.198.1:36769)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.198.2@KRBTEST.COM" created.
I20260925 22:47:36.753861 26436 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.198.1@KRBTEST.COM'} at 127.10.198.1:47949
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.198.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376259650642-11032-0/krb5kdc/kudu_127.10.198.2.keytab.
Entry for principal kudu/127.10.198.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376259650642-11032-0/krb5kdc/kudu_127.10.198.2.keytab.
I20260925 22:47:36.770713 11032 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrWXm8X/build/debug/bin/kudu
/tmp/dist-test-taskrWXm8X/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376259650642-11032-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376259650642-11032-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376259650642-11032-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376259650642-11032-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.198.2:0
--local_ip_for_outbound_sockets=127.10.198.2
--webserver_interface=127.10.198.2
--webserver_port=0
--tserver_master_addrs=127.10.198.62:41557
--keytab_file=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376259650642-11032-0/krb5kdc/kudu_127.10.198.2.keytab
--principal=kudu/127.10.198.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.198.20:41079
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376259650642-11032-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376259650642-11032-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376259650642-11032-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:47:36.889379 26632 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:47:36.889716 26632 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:47:36.889765 26632 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:47:36.894086 26632 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:47:36.894168 26632 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:47:36.894232 26632 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.198.2
I20260925 22:47:36.900489 26632 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.198.20:41079
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376259650642-11032-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376259650642-11032-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376259650642-11032-0/krb5kdc/kudu_127.10.198.2.keytab
--principal=kudu/127.10.198.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.198.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376259650642-11032-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.198.2
--webserver_port=0
--tserver_master_addrs=127.10.198.62:41557
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26632
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.198.2
--log_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376259650642-11032-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:09:16 UTC on ccycloud.quasar-afmhyw.root.comops.site
build id 4673379
I20260925 22:47:36.901645 26632 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:47:36.902698 26632 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:47:36.913848 26638 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:47:36.913866 26637 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:47:36.913925 26640 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:47:36.914202 26632 server_base.cc:1034] running on GCE node
I20260925 22:47:36.914764 26632 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:47:36.915567 26632 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:47:36.916775 26632 hybrid_clock.cc:648] HybridClock initialized: now 1790376456916749 us; error 37 us; skew 500 ppm
Sep 25 22:47:36 dist-test-slave-5vdr krb5kdc[26391](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376456, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.198.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:47:36.920156 26632 init.cc:377] Logged in from keytab as kudu/127.10.198.2@KRBTEST.COM (short username kudu)
I20260925 22:47:36.921253 26632 webserver.cc:460] Webserver started at http://127.10.198.2:40329/ using document root <none> and password file <none>
I20260925 22:47:36.921927 26632 fs_manager.cc:359] Metadata directory not provided
I20260925 22:47:36.922039 26632 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:47:36.922461 26632 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:47:36.924953 26632 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376259650642-11032-0/minicluster-data/ts-1/data/instance:
uuid: "1b6eadd191934a76ba315f4c414dd647"
format_stamp: "Formatted at 2026-09-25 22:47:36 on dist-test-slave-5vdr"
server_key: "5f931a5901e7de1154c19256f2a62735"
server_key_iv: "38c9689b483c352210f94e845e378f22"
server_key_version: "encryptionkey@0"
I20260925 22:47:36.925673 26632 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376259650642-11032-0/minicluster-data/ts-1/wal/instance:
uuid: "1b6eadd191934a76ba315f4c414dd647"
format_stamp: "Formatted at 2026-09-25 22:47:36 on dist-test-slave-5vdr"
server_key: "5f931a5901e7de1154c19256f2a62735"
server_key_iv: "38c9689b483c352210f94e845e378f22"
server_key_version: "encryptionkey@0"
I20260925 22:47:36.930092 26632 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260925 22:47:36.933084 26647 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:47:36.934512 26632 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 22:47:36.934705 26632 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376259650642-11032-0/minicluster-data/ts-1/data,/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376259650642-11032-0/minicluster-data/ts-1/wal
uuid: "1b6eadd191934a76ba315f4c414dd647"
format_stamp: "Formatted at 2026-09-25 22:47:36 on dist-test-slave-5vdr"
server_key: "5f931a5901e7de1154c19256f2a62735"
server_key_iv: "38c9689b483c352210f94e845e378f22"
server_key_version: "encryptionkey@0"
I20260925 22:47:36.934854 26632 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376259650642-11032-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376259650642-11032-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376259650642-11032-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:47:36.946404 26632 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:47:36.949853 26632 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:47:36.950093 26632 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:47:36.950749 26632 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 22:47:36.951669 26632 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:47:36.951749 26632 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:47:36.951828 26632 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:47:36.951922 26632 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.001s
I20260925 22:47:36.962812 26632 rpc_server.cc:307] RPC server started. Bound to: 127.10.198.2:40709
I20260925 22:47:36.962832 26760 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.198.2:40709 every 8 connection(s)
I20260925 22:47:36.963984 26632 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376259650642-11032-0/minicluster-data/ts-1/data/info.pb
I20260925 22:47:36.968110 11032 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrWXm8X/build/debug/bin/kudu as pid 26632
I20260925 22:47:36.968240 11032 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376259650642-11032-0/minicluster-data/ts-1/wal/instance
I20260925 22:47:36.968533 11032 external_mini_cluster.cc:1428] Setting key 75b930732bcdf43b7eebb87cd88c0d1f
Sep 25 22:47:36 dist-test-slave-5vdr krb5kdc[26391](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376456, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.198.2@KRBTEST.COM for kudu/127.10.198.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.10.198.3@KRBTEST.COM; defaulting to no policy
I20260925 22:47:36.980563 26475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:47:36.966033 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.198.2:54525 (local address 127.10.198.62:41557)
0925 22:47:36.966192 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0925 22:47:36.966197 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:47:36.967186 (+ 989us) server_negotiation.cc:407] Connection header received
0925 22:47:36.968906 (+ 1720us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:47:36.968909 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:47:36.968999 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:47:36.969109 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:47:36.970902 (+ 1793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:47:36.971473 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:47:36.972961 (+ 1488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:47:36.973231 (+ 270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:47:36.973364 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:47:36.976242 (+ 2878us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:47:36.976246 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:47:36.976248 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:47:36.976277 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:47:36.977898 (+ 1621us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:47:36.978610 (+ 712us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:47:36.978613 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:47:36.978615 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:47:36.978704 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:47:36.979111 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:47:36.979114 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:47:36.979115 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:47:36.979410 (+ 295us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:47:36.979528 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0925 22:47:36.980237 (+ 709us) server_negotiation.cc:299] Negotiation successful
0925 22:47:36.980391 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.198.3@KRBTEST.COM" created.
I20260925 22:47:36.981739 26763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:47:36.966505 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.198.62:41557 (local address 127.10.198.2:54525)
0925 22:47:36.967002 (+ 497us) negotiation.cc:107] Waiting for socket to connect
0925 22:47:36.967052 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0925 22:47:36.968620 (+ 1568us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:47:36.969366 (+ 746us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:47:36.969375 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:47:36.969878 (+ 503us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:47:36.970688 (+ 810us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:47:36.970715 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:47:36.971660 (+ 945us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:47:36.971666 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:47:36.972801 (+ 1135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:47:36.972814 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:47:36.973419 (+ 605us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:47:36.973423 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:47:36.973626 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:47:36.974365 (+ 739us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:47:36.974384 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:47:36.976015 (+ 1631us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:47:36.978069 (+ 2054us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:47:36.978077 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:47:36.978099 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:47:36.978468 (+ 369us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:47:36.978809 (+ 341us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:47:36.978812 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:47:36.978815 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:47:36.978983 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:47:36.979534 (+ 551us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:47:36.979545 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:47:36.979897 (+ 352us) client_negotiation.cc:769] Sending connection context
0925 22:47:36.980200 (+ 303us) client_negotiation.cc:240] Negotiation successful
0925 22:47:36.980588 (+ 388us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":291,"thread_start_us":126,"threads_started":1}
I20260925 22:47:36.983184 26761 heartbeater.cc:347] Connected to a master server at 127.10.198.62:41557
I20260925 22:47:36.983529 26761 heartbeater.cc:464] Registering TS with master...
I20260925 22:47:36.984272 26761 heartbeater.cc:511] Master 127.10.198.62:41557 requested a full tablet report, sending...
I20260925 22:47:36.985672 26436 ts_manager.cc:194] Registered new tserver with Master: 1b6eadd191934a76ba315f4c414dd647 (127.10.198.2:40709)
I20260925 22:47:36.986586 26436 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.198.2@KRBTEST.COM'} at 127.10.198.2:54525
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.198.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376259650642-11032-0/krb5kdc/kudu_127.10.198.3.keytab.
Entry for principal kudu/127.10.198.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376259650642-11032-0/krb5kdc/kudu_127.10.198.3.keytab.
I20260925 22:47:36.998766 11032 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrWXm8X/build/debug/bin/kudu
/tmp/dist-test-taskrWXm8X/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376259650642-11032-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376259650642-11032-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376259650642-11032-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376259650642-11032-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.198.3:0
--local_ip_for_outbound_sockets=127.10.198.3
--webserver_interface=127.10.198.3
--webserver_port=0
--tserver_master_addrs=127.10.198.62:41557
--keytab_file=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376259650642-11032-0/krb5kdc/kudu_127.10.198.3.keytab
--principal=kudu/127.10.198.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.198.20:41079
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376259650642-11032-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376259650642-11032-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376259650642-11032-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:47:37.121466 26766 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:47:37.121809 26766 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:47:37.121888 26766 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:47:37.127063 26766 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:47:37.127174 26766 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:47:37.127244 26766 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.198.3
I20260925 22:47:37.133893 26766 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.198.20:41079
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376259650642-11032-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376259650642-11032-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376259650642-11032-0/krb5kdc/kudu_127.10.198.3.keytab
--principal=kudu/127.10.198.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.198.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376259650642-11032-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.198.3
--webserver_port=0
--tserver_master_addrs=127.10.198.62:41557
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26766
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.198.3
--log_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376259650642-11032-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:09:16 UTC on ccycloud.quasar-afmhyw.root.comops.site
build id 4673379
I20260925 22:47:37.135156 26766 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:47:37.136406 26766 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:47:37.147198 26772 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:47:37.147198 26774 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:47:37.147202 26771 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:47:37.147353 26766 server_base.cc:1034] running on GCE node
I20260925 22:47:37.148094 26766 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:47:37.148838 26766 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:47:37.150050 26766 hybrid_clock.cc:648] HybridClock initialized: now 1790376457150017 us; error 57 us; skew 500 ppm
Sep 25 22:47:37 dist-test-slave-5vdr krb5kdc[26391](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376457, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.198.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:47:37.153427 26766 init.cc:377] Logged in from keytab as kudu/127.10.198.3@KRBTEST.COM (short username kudu)
I20260925 22:47:37.154489 26766 webserver.cc:460] Webserver started at http://127.10.198.3:43119/ using document root <none> and password file <none>
I20260925 22:47:37.155138 26766 fs_manager.cc:359] Metadata directory not provided
I20260925 22:47:37.155254 26766 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:47:37.155531 26766 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:47:37.158165 26766 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376259650642-11032-0/minicluster-data/ts-2/data/instance:
uuid: "e299c7a0d9b04d1a8d30de149122fc48"
format_stamp: "Formatted at 2026-09-25 22:47:37 on dist-test-slave-5vdr"
server_key: "5e2b92cea1a43418998f5092402082b4"
server_key_iv: "8e5ef74a1b33c58e90f390b421cc177c"
server_key_version: "encryptionkey@0"
I20260925 22:47:37.158934 26766 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376259650642-11032-0/minicluster-data/ts-2/wal/instance:
uuid: "e299c7a0d9b04d1a8d30de149122fc48"
format_stamp: "Formatted at 2026-09-25 22:47:37 on dist-test-slave-5vdr"
server_key: "5e2b92cea1a43418998f5092402082b4"
server_key_iv: "8e5ef74a1b33c58e90f390b421cc177c"
server_key_version: "encryptionkey@0"
I20260925 22:47:37.163494 26766 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260925 22:47:37.166692 26781 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:47:37.168331 26766 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 22:47:37.168548 26766 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376259650642-11032-0/minicluster-data/ts-2/data,/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376259650642-11032-0/minicluster-data/ts-2/wal
uuid: "e299c7a0d9b04d1a8d30de149122fc48"
format_stamp: "Formatted at 2026-09-25 22:47:37 on dist-test-slave-5vdr"
server_key: "5e2b92cea1a43418998f5092402082b4"
server_key_iv: "8e5ef74a1b33c58e90f390b421cc177c"
server_key_version: "encryptionkey@0"
I20260925 22:47:37.168715 26766 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376259650642-11032-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376259650642-11032-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376259650642-11032-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:47:37.185715 26766 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:47:37.190157 26766 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:47:37.190442 26766 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:47:37.191177 26766 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 22:47:37.192625 26766 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:47:37.192709 26766 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:47:37.192781 26766 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:47:37.192808 26766 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 22:47:37.204147 26766 rpc_server.cc:307] RPC server started. Bound to: 127.10.198.3:43405
I20260925 22:47:37.204166 26894 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.198.3:43405 every 8 connection(s)
I20260925 22:47:37.205232 26766 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376259650642-11032-0/minicluster-data/ts-2/data/info.pb
I20260925 22:47:37.206684 11032 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrWXm8X/build/debug/bin/kudu as pid 26766
I20260925 22:47:37.206831 11032 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376259650642-11032-0/minicluster-data/ts-2/wal/instance
I20260925 22:47:37.207144 11032 external_mini_cluster.cc:1428] Setting key 7401b8e48b8e1e32b3a57ab86a0aa89e
Sep 25 22:47:37 dist-test-slave-5vdr krb5kdc[26391](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376457, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.198.3@KRBTEST.COM for kudu/127.10.198.62@KRBTEST.COM
I20260925 22:47:37.221361 26475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:47:37.207464 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.198.3:35301 (local address 127.10.198.62:41557)
0925 22:47:37.207663 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0925 22:47:37.207669 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 22:47:37.208865 (+ 1196us) server_negotiation.cc:407] Connection header received
0925 22:47:37.210315 (+ 1450us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:47:37.210318 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:47:37.210374 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:47:37.210460 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:47:37.211787 (+ 1327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:47:37.212425 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:47:37.213688 (+ 1263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:47:37.213915 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:47:37.214016 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:47:37.217085 (+ 3069us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:47:37.217089 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:47:37.217091 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:47:37.217122 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:47:37.218679 (+ 1557us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:47:37.219426 (+ 747us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:47:37.219429 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:47:37.219430 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:47:37.219521 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:47:37.219993 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:47:37.219996 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:47:37.219997 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:47:37.220208 (+ 211us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:47:37.220346 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0925 22:47:37.221036 (+ 690us) server_negotiation.cc:299] Negotiation successful
0925 22:47:37.221190 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260925 22:47:37.222772 26897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:47:37.208113 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.198.62:41557 (local address 127.10.198.3:35301)
0925 22:47:37.208698 (+ 585us) negotiation.cc:107] Waiting for socket to connect
0925 22:47:37.208759 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0925 22:47:37.210045 (+ 1286us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:47:37.210651 (+ 606us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:47:37.210659 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:47:37.211084 (+ 425us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:47:37.211631 (+ 547us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:47:37.211646 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:47:37.212567 (+ 921us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:47:37.212572 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:47:37.213530 (+ 958us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:47:37.213542 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:47:37.214084 (+ 542us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:47:37.214087 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:47:37.214245 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:47:37.215066 (+ 821us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:47:37.215086 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:47:37.216853 (+ 1767us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:47:37.218853 (+ 2000us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:47:37.218861 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:47:37.218921 (+ 60us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:47:37.219312 (+ 391us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:47:37.219662 (+ 350us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:47:37.219665 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:47:37.219666 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:47:37.219828 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:47:37.220347 (+ 519us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:47:37.220356 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:47:37.220713 (+ 357us) client_negotiation.cc:769] Sending connection context
0925 22:47:37.220975 (+ 262us) client_negotiation.cc:240] Negotiation successful
0925 22:47:37.221384 (+ 409us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":349,"thread_start_us":160,"threads_started":1}
I20260925 22:47:37.224515 26895 heartbeater.cc:347] Connected to a master server at 127.10.198.62:41557
I20260925 22:47:37.224848 26895 heartbeater.cc:464] Registering TS with master...
I20260925 22:47:37.225462 26895 heartbeater.cc:511] Master 127.10.198.62:41557 requested a full tablet report, sending...
I20260925 22:47:37.227072 26436 ts_manager.cc:194] Registered new tserver with Master: e299c7a0d9b04d1a8d30de149122fc48 (127.10.198.3:43405)
I20260925 22:47:37.228060 26436 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.198.3@KRBTEST.COM'} at 127.10.198.3:35301
I20260925 22:47:37.233875 11032 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 22:47:37.244751 26475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:47:37.236393 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53020 (local address 127.10.198.62:41557)
0925 22:47:37.236552 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0925 22:47:37.236559 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 22:47:37.236629 (+ 70us) server_negotiation.cc:407] Connection header received
0925 22:47:37.236825 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:47:37.236828 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:47:37.236874 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:47:37.236961 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:47:37.237710 (+ 749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:47:37.238237 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:47:37.239268 (+ 1031us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:47:37.239494 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:47:37.239644 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:47:37.241199 (+ 1555us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:47:37.241207 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:47:37.241214 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:47:37.241245 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:47:37.242776 (+ 1531us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:47:37.243287 (+ 511us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:47:37.243291 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:47:37.243295 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:47:37.243385 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:47:37.243708 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:47:37.243713 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:47:37.243714 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:47:37.244038 (+ 324us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:47:37.244185 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0925 22:47:37.244438 (+ 253us) server_negotiation.cc:299] Negotiation successful
0925 22:47:37.244569 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260925 22:47:37.248406 26436 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53020:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 22:47:37.251613 26436 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 22:47:37.268129 26906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:47:37.261555 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.198.1:36769 (local address 127.0.0.1:42790)
0925 22:47:37.262081 (+ 526us) negotiation.cc:107] Waiting for socket to connect
0925 22:47:37.262143 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0925 22:47:37.262293 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:47:37.263361 (+ 1068us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:47:37.263369 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:47:37.263397 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:47:37.263567 (+ 170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:47:37.263599 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:47:37.264914 (+ 1315us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:47:37.264921 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:47:37.266108 (+ 1187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:47:37.266118 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:47:37.267426 (+ 1308us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:47:37.267429 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:47:37.267570 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:47:37.267621 (+ 51us) client_negotiation.cc:769] Sending connection context
0925 22:47:37.267755 (+ 134us) client_negotiation.cc:240] Negotiation successful
0925 22:47:37.267952 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":328,"thread_start_us":78,"threads_started":1}
I20260925 22:47:37.268167 26907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:47:37.261694 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42790 (local address 127.10.198.1:36769)
0925 22:47:37.262123 (+ 429us) server_negotiation.cc:206] Beginning negotiation
0925 22:47:37.262131 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 22:47:37.262224 (+ 93us) server_negotiation.cc:407] Connection header received
0925 22:47:37.262523 (+ 299us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:47:37.262529 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:47:37.262903 (+ 374us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:47:37.263084 (+ 181us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:47:37.263727 (+ 643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:47:37.264790 (+ 1063us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:47:37.266277 (+ 1487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:47:37.267017 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:47:37.267158 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:47:37.267346 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0925 22:47:37.267823 (+ 477us) server_negotiation.cc:299] Negotiation successful
0925 22:47:37.267990 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":319,"thread_start_us":130,"threads_started":1}
I20260925 22:47:37.268512 26909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:47:37.262705 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.198.2:40709 (local address 127.0.0.1:33516)
0925 22:47:37.263293 (+ 588us) negotiation.cc:107] Waiting for socket to connect
0925 22:47:37.263307 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 22:47:37.263498 (+ 191us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:47:37.264229 (+ 731us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:47:37.264232 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:47:37.264250 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:47:37.264332 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:47:37.264338 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:47:37.265746 (+ 1408us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:47:37.265749 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:47:37.266848 (+ 1099us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:47:37.266858 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:47:37.268109 (+ 1251us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:47:37.268113 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:47:37.268217 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:47:37.268243 (+ 26us) client_negotiation.cc:769] Sending connection context
0925 22:47:37.268323 (+ 80us) client_negotiation.cc:240] Negotiation successful
0925 22:47:37.268407 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":499,"thread_start_us":90,"threads_started":1}
I20260925 22:47:37.268872 26910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:47:37.262582 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41702 (local address 127.10.198.3:43405)
0925 22:47:37.263483 (+ 901us) server_negotiation.cc:206] Beginning negotiation
0925 22:47:37.263495 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 22:47:37.263521 (+ 26us) server_negotiation.cc:407] Connection header received
0925 22:47:37.263598 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:47:37.263604 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:47:37.263815 (+ 211us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:47:37.264030 (+ 215us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:47:37.264257 (+ 227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:47:37.265397 (+ 1140us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:47:37.266978 (+ 1581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:47:37.267874 (+ 896us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:47:37.267974 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:47:37.268104 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0925 22:47:37.268588 (+ 484us) server_negotiation.cc:299] Negotiation successful
0925 22:47:37.268714 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":797,"thread_start_us":185,"threads_started":1}
I20260925 22:47:37.269200 26911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:47:37.262872 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33516 (local address 127.10.198.2:40709)
0925 22:47:37.263525 (+ 653us) server_negotiation.cc:206] Beginning negotiation
0925 22:47:37.263543 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0925 22:47:37.263584 (+ 41us) server_negotiation.cc:407] Connection header received
0925 22:47:37.263664 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:47:37.263669 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:47:37.263901 (+ 232us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:47:37.264049 (+ 148us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:47:37.264462 (+ 413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:47:37.265609 (+ 1147us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:47:37.266989 (+ 1380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:47:37.267714 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:47:37.267904 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:47:37.268044 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0925 22:47:37.268831 (+ 787us) server_negotiation.cc:299] Negotiation successful
0925 22:47:37.268987 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":460,"thread_start_us":119,"threads_started":1}
I20260925 22:47:37.268674 26908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:47:37.262351 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.198.3:43405 (local address 127.0.0.1:41702)
0925 22:47:37.263042 (+ 691us) negotiation.cc:107] Waiting for socket to connect
0925 22:47:37.263059 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 22:47:37.263155 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:47:37.264000 (+ 845us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:47:37.264003 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:47:37.264022 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:47:37.264126 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:47:37.264133 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:47:37.265549 (+ 1416us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:47:37.265552 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:47:37.266811 (+ 1259us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:47:37.266823 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:47:37.268172 (+ 1349us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:47:37.268176 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:47:37.268290 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:47:37.268327 (+ 37us) client_negotiation.cc:769] Sending connection context
0925 22:47:37.268398 (+ 71us) client_negotiation.cc:240] Negotiation successful
0925 22:47:37.268467 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":582,"thread_start_us":104,"threads_started":1}
I20260925 22:47:37.272010 26695 tablet_service.cc:1467] Processing CreateTablet for tablet 1220ed2f8c4a4138b6b4091903e8b418 (DEFAULT_TABLE table=test-table [id=aef2b90a17464ff58985620109391cb2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 22:47:37.271972 26561 tablet_service.cc:1467] Processing CreateTablet for tablet 1220ed2f8c4a4138b6b4091903e8b418 (DEFAULT_TABLE table=test-table [id=aef2b90a17464ff58985620109391cb2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 22:47:37.272351 26829 tablet_service.cc:1467] Processing CreateTablet for tablet 1220ed2f8c4a4138b6b4091903e8b418 (DEFAULT_TABLE table=test-table [id=aef2b90a17464ff58985620109391cb2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 22:47:37.273329 26829 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1220ed2f8c4a4138b6b4091903e8b418. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:47:37.273329 26561 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1220ed2f8c4a4138b6b4091903e8b418. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:47:37.274014 26695 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1220ed2f8c4a4138b6b4091903e8b418. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:47:37.281291 26912 tablet_bootstrap.cc:492] T 1220ed2f8c4a4138b6b4091903e8b418 P 1b6eadd191934a76ba315f4c414dd647: Bootstrap starting.
I20260925 22:47:37.281791 26913 tablet_bootstrap.cc:492] T 1220ed2f8c4a4138b6b4091903e8b418 P 84c93dfd42c94d188c85a636289a4e23: Bootstrap starting.
I20260925 22:47:37.282217 26914 tablet_bootstrap.cc:492] T 1220ed2f8c4a4138b6b4091903e8b418 P e299c7a0d9b04d1a8d30de149122fc48: Bootstrap starting.
I20260925 22:47:37.285033 26914 tablet_bootstrap.cc:654] T 1220ed2f8c4a4138b6b4091903e8b418 P e299c7a0d9b04d1a8d30de149122fc48: Neither blocks nor log segments found. Creating new log.
I20260925 22:47:37.286201 26913 tablet_bootstrap.cc:654] T 1220ed2f8c4a4138b6b4091903e8b418 P 84c93dfd42c94d188c85a636289a4e23: Neither blocks nor log segments found. Creating new log.
I20260925 22:47:37.286309 26914 log.cc:824] T 1220ed2f8c4a4138b6b4091903e8b418 P e299c7a0d9b04d1a8d30de149122fc48: Log is configured to *not* fsync() on all Append() calls
I20260925 22:47:37.287038 26913 log.cc:824] T 1220ed2f8c4a4138b6b4091903e8b418 P 84c93dfd42c94d188c85a636289a4e23: Log is configured to *not* fsync() on all Append() calls
I20260925 22:47:37.287698 26912 tablet_bootstrap.cc:654] T 1220ed2f8c4a4138b6b4091903e8b418 P 1b6eadd191934a76ba315f4c414dd647: Neither blocks nor log segments found. Creating new log.
I20260925 22:47:37.288527 26914 tablet_bootstrap.cc:492] T 1220ed2f8c4a4138b6b4091903e8b418 P e299c7a0d9b04d1a8d30de149122fc48: No bootstrap required, opened a new log
I20260925 22:47:37.288626 26912 log.cc:824] T 1220ed2f8c4a4138b6b4091903e8b418 P 1b6eadd191934a76ba315f4c414dd647: Log is configured to *not* fsync() on all Append() calls
I20260925 22:47:37.288794 26914 ts_tablet_manager.cc:1397] T 1220ed2f8c4a4138b6b4091903e8b418 P e299c7a0d9b04d1a8d30de149122fc48: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260925 22:47:37.289335 26913 tablet_bootstrap.cc:492] T 1220ed2f8c4a4138b6b4091903e8b418 P 84c93dfd42c94d188c85a636289a4e23: No bootstrap required, opened a new log
I20260925 22:47:37.289563 26913 ts_tablet_manager.cc:1397] T 1220ed2f8c4a4138b6b4091903e8b418 P 84c93dfd42c94d188c85a636289a4e23: Time spent bootstrapping tablet: real 0.008s user 0.004s sys 0.000s
I20260925 22:47:37.290771 26912 tablet_bootstrap.cc:492] T 1220ed2f8c4a4138b6b4091903e8b418 P 1b6eadd191934a76ba315f4c414dd647: No bootstrap required, opened a new log
I20260925 22:47:37.290983 26912 ts_tablet_manager.cc:1397] T 1220ed2f8c4a4138b6b4091903e8b418 P 1b6eadd191934a76ba315f4c414dd647: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.000s
I20260925 22:47:37.292367 26914 raft_consensus.cc:348] T 1220ed2f8c4a4138b6b4091903e8b418 P e299c7a0d9b04d1a8d30de149122fc48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84c93dfd42c94d188c85a636289a4e23" member_type: VOTER last_known_addr { host: "127.10.198.1" port: 36769 } } peers { permanent_uuid: "e299c7a0d9b04d1a8d30de149122fc48" member_type: VOTER last_known_addr { host: "127.10.198.3" port: 43405 } } peers { permanent_uuid: "1b6eadd191934a76ba315f4c414dd647" member_type: VOTER last_known_addr { host: "127.10.198.2" port: 40709 } }
I20260925 22:47:37.292827 26914 raft_consensus.cc:374] T 1220ed2f8c4a4138b6b4091903e8b418 P e299c7a0d9b04d1a8d30de149122fc48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:47:37.292948 26914 raft_consensus.cc:729] T 1220ed2f8c4a4138b6b4091903e8b418 P e299c7a0d9b04d1a8d30de149122fc48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e299c7a0d9b04d1a8d30de149122fc48, State: Initialized, Role: FOLLOWER
I20260925 22:47:37.293551 26914 consensus_queue.cc:260] T 1220ed2f8c4a4138b6b4091903e8b418 P e299c7a0d9b04d1a8d30de149122fc48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84c93dfd42c94d188c85a636289a4e23" member_type: VOTER last_known_addr { host: "127.10.198.1" port: 36769 } } peers { permanent_uuid: "e299c7a0d9b04d1a8d30de149122fc48" member_type: VOTER last_known_addr { host: "127.10.198.3" port: 43405 } } peers { permanent_uuid: "1b6eadd191934a76ba315f4c414dd647" member_type: VOTER last_known_addr { host: "127.10.198.2" port: 40709 } }
I20260925 22:47:37.294340 26895 heartbeater.cc:503] Master 127.10.198.62:41557 was elected leader, sending a full tablet report...
I20260925 22:47:37.294684 26914 ts_tablet_manager.cc:1428] T 1220ed2f8c4a4138b6b4091903e8b418 P e299c7a0d9b04d1a8d30de149122fc48: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260925 22:47:37.294862 26913 raft_consensus.cc:348] T 1220ed2f8c4a4138b6b4091903e8b418 P 84c93dfd42c94d188c85a636289a4e23 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84c93dfd42c94d188c85a636289a4e23" member_type: VOTER last_known_addr { host: "127.10.198.1" port: 36769 } } peers { permanent_uuid: "e299c7a0d9b04d1a8d30de149122fc48" member_type: VOTER last_known_addr { host: "127.10.198.3" port: 43405 } } peers { permanent_uuid: "1b6eadd191934a76ba315f4c414dd647" member_type: VOTER last_known_addr { host: "127.10.198.2" port: 40709 } }
I20260925 22:47:37.295238 26913 raft_consensus.cc:374] T 1220ed2f8c4a4138b6b4091903e8b418 P 84c93dfd42c94d188c85a636289a4e23 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:47:37.295068 26912 raft_consensus.cc:348] T 1220ed2f8c4a4138b6b4091903e8b418 P 1b6eadd191934a76ba315f4c414dd647 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84c93dfd42c94d188c85a636289a4e23" member_type: VOTER last_known_addr { host: "127.10.198.1" port: 36769 } } peers { permanent_uuid: "e299c7a0d9b04d1a8d30de149122fc48" member_type: VOTER last_known_addr { host: "127.10.198.3" port: 43405 } } peers { permanent_uuid: "1b6eadd191934a76ba315f4c414dd647" member_type: VOTER last_known_addr { host: "127.10.198.2" port: 40709 } }
I20260925 22:47:37.295306 26913 raft_consensus.cc:729] T 1220ed2f8c4a4138b6b4091903e8b418 P 84c93dfd42c94d188c85a636289a4e23 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 84c93dfd42c94d188c85a636289a4e23, State: Initialized, Role: FOLLOWER
I20260925 22:47:37.295357 26912 raft_consensus.cc:374] T 1220ed2f8c4a4138b6b4091903e8b418 P 1b6eadd191934a76ba315f4c414dd647 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:47:37.295429 26912 raft_consensus.cc:729] T 1220ed2f8c4a4138b6b4091903e8b418 P 1b6eadd191934a76ba315f4c414dd647 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b6eadd191934a76ba315f4c414dd647, State: Initialized, Role: FOLLOWER
I20260925 22:47:37.295950 26912 consensus_queue.cc:260] T 1220ed2f8c4a4138b6b4091903e8b418 P 1b6eadd191934a76ba315f4c414dd647 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84c93dfd42c94d188c85a636289a4e23" member_type: VOTER last_known_addr { host: "127.10.198.1" port: 36769 } } peers { permanent_uuid: "e299c7a0d9b04d1a8d30de149122fc48" member_type: VOTER last_known_addr { host: "127.10.198.3" port: 43405 } } peers { permanent_uuid: "1b6eadd191934a76ba315f4c414dd647" member_type: VOTER last_known_addr { host: "127.10.198.2" port: 40709 } }
I20260925 22:47:37.296015 26913 consensus_queue.cc:260] T 1220ed2f8c4a4138b6b4091903e8b418 P 84c93dfd42c94d188c85a636289a4e23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84c93dfd42c94d188c85a636289a4e23" member_type: VOTER last_known_addr { host: "127.10.198.1" port: 36769 } } peers { permanent_uuid: "e299c7a0d9b04d1a8d30de149122fc48" member_type: VOTER last_known_addr { host: "127.10.198.3" port: 43405 } } peers { permanent_uuid: "1b6eadd191934a76ba315f4c414dd647" member_type: VOTER last_known_addr { host: "127.10.198.2" port: 40709 } }
I20260925 22:47:37.296758 26761 heartbeater.cc:503] Master 127.10.198.62:41557 was elected leader, sending a full tablet report...
I20260925 22:47:37.297147 26912 ts_tablet_manager.cc:1428] T 1220ed2f8c4a4138b6b4091903e8b418 P 1b6eadd191934a76ba315f4c414dd647: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260925 22:47:37.298048 26627 heartbeater.cc:503] Master 127.10.198.62:41557 was elected leader, sending a full tablet report...
I20260925 22:47:37.298063 26913 ts_tablet_manager.cc:1428] T 1220ed2f8c4a4138b6b4091903e8b418 P 84c93dfd42c94d188c85a636289a4e23: Time spent starting tablet: real 0.008s user 0.008s sys 0.000s
W20260925 22:47:37.456251 26896 tablet.cc:2369] T 1220ed2f8c4a4138b6b4091903e8b418 P e299c7a0d9b04d1a8d30de149122fc48: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 22:47:37.465315 26762 tablet.cc:2369] T 1220ed2f8c4a4138b6b4091903e8b418 P 1b6eadd191934a76ba315f4c414dd647: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 22:47:37.475706 26919 raft_consensus.cc:482] T 1220ed2f8c4a4138b6b4091903e8b418 P 1b6eadd191934a76ba315f4c414dd647 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 22:47:37.476009 26919 raft_consensus.cc:504] T 1220ed2f8c4a4138b6b4091903e8b418 P 1b6eadd191934a76ba315f4c414dd647 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84c93dfd42c94d188c85a636289a4e23" member_type: VOTER last_known_addr { host: "127.10.198.1" port: 36769 } } peers { permanent_uuid: "e299c7a0d9b04d1a8d30de149122fc48" member_type: VOTER last_known_addr { host: "127.10.198.3" port: 43405 } } peers { permanent_uuid: "1b6eadd191934a76ba315f4c414dd647" member_type: VOTER last_known_addr { host: "127.10.198.2" port: 40709 } }
I20260925 22:47:37.477211 26919 leader_election.cc:288] T 1220ed2f8c4a4138b6b4091903e8b418 P 1b6eadd191934a76ba315f4c414dd647 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 84c93dfd42c94d188c85a636289a4e23 (127.10.198.1:36769), e299c7a0d9b04d1a8d30de149122fc48 (127.10.198.3:43405)
W20260925 22:47:37.481654 26628 tablet.cc:2369] T 1220ed2f8c4a4138b6b4091903e8b418 P 84c93dfd42c94d188c85a636289a4e23: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 22:47:37.481765 26763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:47:37.477550 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.198.1:36769 (local address 127.10.198.2:47093)
0925 22:47:37.477689 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0925 22:47:37.477708 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0925 22:47:37.477815 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:47:37.478225 (+ 410us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:47:37.478228 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:47:37.478298 (+ 70us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:47:37.478448 (+ 150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:47:37.478456 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:47:37.479384 (+ 928us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:47:37.479388 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:47:37.480429 (+ 1041us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:47:37.480442 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:47:37.481271 (+ 829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:47:37.481275 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:47:37.481391 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:47:37.481430 (+ 39us) client_negotiation.cc:769] Sending connection context
0925 22:47:37.481505 (+ 75us) client_negotiation.cc:240] Negotiation successful
0925 22:47:37.481601 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":54,"mutex_wait_us":1,"spinlock_wait_cycles":21632}
I20260925 22:47:37.481861 26907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:47:37.477727 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.198.2:47093 (local address 127.10.198.1:36769)
0925 22:47:37.477864 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0925 22:47:37.477869 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:47:37.477888 (+ 19us) server_negotiation.cc:407] Connection header received
0925 22:47:37.477961 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:47:37.477963 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:47:37.478034 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:47:37.478144 (+ 110us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:47:37.478589 (+ 445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:47:37.479232 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:47:37.480583 (+ 1351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:47:37.481137 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:47:37.481245 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:47:37.481292 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0925 22:47:37.481645 (+ 353us) server_negotiation.cc:299] Negotiation successful
0925 22:47:37.481735 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260925 22:47:37.481994 26910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:47:37.478025 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.198.2:54447 (local address 127.10.198.3:43405)
0925 22:47:37.478161 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0925 22:47:37.478164 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 22:47:37.478273 (+ 109us) server_negotiation.cc:407] Connection header received
0925 22:47:37.478421 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:47:37.478424 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:47:37.478472 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:47:37.478547 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:47:37.478863 (+ 316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:47:37.479587 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:47:37.480869 (+ 1282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:47:37.481366 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:47:37.481510 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:47:37.481557 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0925 22:47:37.481785 (+ 228us) server_negotiation.cc:299] Negotiation successful
0925 22:47:37.481876 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260925 22:47:37.482401 26921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:47:37.477860 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.198.3:43405 (local address 127.10.198.2:54447)
0925 22:47:37.478162 (+ 302us) negotiation.cc:107] Waiting for socket to connect
0925 22:47:37.478178 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 22:47:37.478316 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:47:37.478597 (+ 281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:47:37.478600 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:47:37.478618 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:47:37.478734 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:47:37.478742 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:47:37.479757 (+ 1015us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:47:37.479760 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:47:37.480721 (+ 961us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:47:37.480733 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:47:37.481506 (+ 773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:47:37.481510 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:47:37.481616 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:47:37.481639 (+ 23us) client_negotiation.cc:769] Sending connection context
0925 22:47:37.482154 (+ 515us) client_negotiation.cc:240] Negotiation successful
0925 22:47:37.482241 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":224,"spinlock_wait_cycles":1280,"thread_start_us":92,"threads_started":1}
I20260925 22:47:37.482514 26581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1220ed2f8c4a4138b6b4091903e8b418" candidate_uuid: "1b6eadd191934a76ba315f4c414dd647" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "84c93dfd42c94d188c85a636289a4e23" is_pre_election: true
I20260925 22:47:37.482899 26581 raft_consensus.cc:2454] T 1220ed2f8c4a4138b6b4091903e8b418 P 84c93dfd42c94d188c85a636289a4e23 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1b6eadd191934a76ba315f4c414dd647 in term 0.
I20260925 22:47:37.483243 26849 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1220ed2f8c4a4138b6b4091903e8b418" candidate_uuid: "1b6eadd191934a76ba315f4c414dd647" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e299c7a0d9b04d1a8d30de149122fc48" is_pre_election: true
I20260925 22:47:37.483490 26648 leader_election.cc:302] T 1220ed2f8c4a4138b6b4091903e8b418 P 1b6eadd191934a76ba315f4c414dd647 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b6eadd191934a76ba315f4c414dd647, 84c93dfd42c94d188c85a636289a4e23; no voters:
I20260925 22:47:37.483597 26849 raft_consensus.cc:2454] T 1220ed2f8c4a4138b6b4091903e8b418 P e299c7a0d9b04d1a8d30de149122fc48 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1b6eadd191934a76ba315f4c414dd647 in term 0.
I20260925 22:47:37.483825 26919 raft_consensus.cc:2781] T 1220ed2f8c4a4138b6b4091903e8b418 P 1b6eadd191934a76ba315f4c414dd647 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 22:47:37.483980 26919 raft_consensus.cc:482] T 1220ed2f8c4a4138b6b4091903e8b418 P 1b6eadd191934a76ba315f4c414dd647 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 22:47:37.484057 26919 raft_consensus.cc:3037] T 1220ed2f8c4a4138b6b4091903e8b418 P 1b6eadd191934a76ba315f4c414dd647 [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:47:37.485203 26919 raft_consensus.cc:504] T 1220ed2f8c4a4138b6b4091903e8b418 P 1b6eadd191934a76ba315f4c414dd647 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84c93dfd42c94d188c85a636289a4e23" member_type: VOTER last_known_addr { host: "127.10.198.1" port: 36769 } } peers { permanent_uuid: "e299c7a0d9b04d1a8d30de149122fc48" member_type: VOTER last_known_addr { host: "127.10.198.3" port: 43405 } } peers { permanent_uuid: "1b6eadd191934a76ba315f4c414dd647" member_type: VOTER last_known_addr { host: "127.10.198.2" port: 40709 } }
I20260925 22:47:37.485675 26919 leader_election.cc:288] T 1220ed2f8c4a4138b6b4091903e8b418 P 1b6eadd191934a76ba315f4c414dd647 [CANDIDATE]: Term 1 election: Requested vote from peers 84c93dfd42c94d188c85a636289a4e23 (127.10.198.1:36769), e299c7a0d9b04d1a8d30de149122fc48 (127.10.198.3:43405)
I20260925 22:47:37.486200 26581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1220ed2f8c4a4138b6b4091903e8b418" candidate_uuid: "1b6eadd191934a76ba315f4c414dd647" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "84c93dfd42c94d188c85a636289a4e23"
I20260925 22:47:37.486218 26849 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1220ed2f8c4a4138b6b4091903e8b418" candidate_uuid: "1b6eadd191934a76ba315f4c414dd647" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e299c7a0d9b04d1a8d30de149122fc48"
I20260925 22:47:37.486369 26581 raft_consensus.cc:3037] T 1220ed2f8c4a4138b6b4091903e8b418 P 84c93dfd42c94d188c85a636289a4e23 [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:47:37.486379 26849 raft_consensus.cc:3037] T 1220ed2f8c4a4138b6b4091903e8b418 P e299c7a0d9b04d1a8d30de149122fc48 [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:47:37.487527 26849 raft_consensus.cc:2454] T 1220ed2f8c4a4138b6b4091903e8b418 P e299c7a0d9b04d1a8d30de149122fc48 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1b6eadd191934a76ba315f4c414dd647 in term 1.
I20260925 22:47:37.487783 26581 raft_consensus.cc:2454] T 1220ed2f8c4a4138b6b4091903e8b418 P 84c93dfd42c94d188c85a636289a4e23 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1b6eadd191934a76ba315f4c414dd647 in term 1.
I20260925 22:47:37.488054 26648 leader_election.cc:302] T 1220ed2f8c4a4138b6b4091903e8b418 P 1b6eadd191934a76ba315f4c414dd647 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b6eadd191934a76ba315f4c414dd647, e299c7a0d9b04d1a8d30de149122fc48; no voters:
I20260925 22:47:37.488324 26919 raft_consensus.cc:2781] T 1220ed2f8c4a4138b6b4091903e8b418 P 1b6eadd191934a76ba315f4c414dd647 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 22:47:37.488591 26919 raft_consensus.cc:686] T 1220ed2f8c4a4138b6b4091903e8b418 P 1b6eadd191934a76ba315f4c414dd647 [term 1 LEADER]: Becoming Leader. State: Replica: 1b6eadd191934a76ba315f4c414dd647, State: Running, Role: LEADER
I20260925 22:47:37.488960 26919 consensus_queue.cc:237] T 1220ed2f8c4a4138b6b4091903e8b418 P 1b6eadd191934a76ba315f4c414dd647 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84c93dfd42c94d188c85a636289a4e23" member_type: VOTER last_known_addr { host: "127.10.198.1" port: 36769 } } peers { permanent_uuid: "e299c7a0d9b04d1a8d30de149122fc48" member_type: VOTER last_known_addr { host: "127.10.198.3" port: 43405 } } peers { permanent_uuid: "1b6eadd191934a76ba315f4c414dd647" member_type: VOTER last_known_addr { host: "127.10.198.2" port: 40709 } }
I20260925 22:47:37.492698 26435 catalog_manager.cc:5495] T 1220ed2f8c4a4138b6b4091903e8b418 P 1b6eadd191934a76ba315f4c414dd647 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1b6eadd191934a76ba315f4c414dd647 (127.10.198.2). New cstate: current_term: 1 leader_uuid: "1b6eadd191934a76ba315f4c414dd647" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84c93dfd42c94d188c85a636289a4e23" member_type: VOTER last_known_addr { host: "127.10.198.1" port: 36769 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e299c7a0d9b04d1a8d30de149122fc48" member_type: VOTER last_known_addr { host: "127.10.198.3" port: 43405 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1b6eadd191934a76ba315f4c414dd647" member_type: VOTER last_known_addr { host: "127.10.198.2" port: 40709 } health_report { overall_health: HEALTHY } } }
I20260925 22:47:37.507299 26911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:47:37.503752 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33532 (local address 127.10.198.2:40709)
0925 22:47:37.503950 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0925 22:47:37.503954 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:47:37.503967 (+ 13us) server_negotiation.cc:407] Connection header received
0925 22:47:37.504059 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:47:37.504062 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:47:37.504118 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:47:37.504219 (+ 101us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 22:47:37.504452 (+ 233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:47:37.504975 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:47:37.505902 (+ 927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:47:37.506150 (+ 248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:47:37.506247 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:47:37.506524 (+ 277us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 22:47:37.506975 (+ 451us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 22:47:37.507075 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0925 22:47:37.507124 (+ 49us) server_negotiation.cc:299] Negotiation successful
0925 22:47:37.507181 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260925 22:47:37.514657 26849 raft_consensus.cc:1261] T 1220ed2f8c4a4138b6b4091903e8b418 P e299c7a0d9b04d1a8d30de149122fc48 [term 1 FOLLOWER]: Refusing update from remote peer 1b6eadd191934a76ba315f4c414dd647: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 22:47:37.514663 26581 raft_consensus.cc:1261] T 1220ed2f8c4a4138b6b4091903e8b418 P 84c93dfd42c94d188c85a636289a4e23 [term 1 FOLLOWER]: Refusing update from remote peer 1b6eadd191934a76ba315f4c414dd647: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 22:47:37.515661 26919 consensus_queue.cc:1035] T 1220ed2f8c4a4138b6b4091903e8b418 P 1b6eadd191934a76ba315f4c414dd647 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e299c7a0d9b04d1a8d30de149122fc48" member_type: VOTER last_known_addr { host: "127.10.198.3" port: 43405 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 22:47:37.515975 26922 consensus_queue.cc:1035] T 1220ed2f8c4a4138b6b4091903e8b418 P 1b6eadd191934a76ba315f4c414dd647 [LEADER]: Connected to new peer: Peer: permanent_uuid: "84c93dfd42c94d188c85a636289a4e23" member_type: VOTER last_known_addr { host: "127.10.198.1" port: 36769 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 22:47:37.526187 26926 mvcc.cc:205] Tried to move back new op lower bound from 7333381969969295360 to 7333381969877618688. Current Snapshot: MvccSnapshot[applied={T|T < 7333381969969295360}]
I20260925 22:47:37.526589 26927 mvcc.cc:205] Tried to move back new op lower bound from 7333381969969295360 to 7333381969877618688. Current Snapshot: MvccSnapshot[applied={T|T < 7333381969969295360}]
I20260925 22:47:37.535678 26435 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53020:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 22:47:37.535914 26435 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53020:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 22:47:37.538938 26435 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3f1861b6350a4355939000f5afa487a5: Sending DeleteTablet for 3 replicas of tablet 1220ed2f8c4a4138b6b4091903e8b418
I20260925 22:47:37.540006 11032 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrWXm8X/build/debug/bin/kudu with pid 26498
I20260925 22:47:37.540190 26829 tablet_service.cc:1514] Processing DeleteTablet for tablet 1220ed2f8c4a4138b6b4091903e8b418 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 22:47:37 UTC) from {username='kudu', principal='kudu/127.10.198.62@KRBTEST.COM'} at 127.0.0.1:41702
I20260925 22:47:37.542943 26935 tablet_replica.cc:330] T 1220ed2f8c4a4138b6b4091903e8b418 P e299c7a0d9b04d1a8d30de149122fc48: stopping tablet replica
I20260925 22:47:37.542871 26695 tablet_service.cc:1514] Processing DeleteTablet for tablet 1220ed2f8c4a4138b6b4091903e8b418 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 22:47:37 UTC) from {username='kudu', principal='kudu/127.10.198.62@KRBTEST.COM'} at 127.0.0.1:33516
I20260925 22:47:37.543395 26935 raft_consensus.cc:2229] T 1220ed2f8c4a4138b6b4091903e8b418 P e299c7a0d9b04d1a8d30de149122fc48 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 22:47:37.543810 26935 raft_consensus.cc:2258] T 1220ed2f8c4a4138b6b4091903e8b418 P e299c7a0d9b04d1a8d30de149122fc48 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 22:47:37.544090 26936 tablet_replica.cc:330] T 1220ed2f8c4a4138b6b4091903e8b418 P 1b6eadd191934a76ba315f4c414dd647: stopping tablet replica
I20260925 22:47:37.544643 26936 raft_consensus.cc:2229] T 1220ed2f8c4a4138b6b4091903e8b418 P 1b6eadd191934a76ba315f4c414dd647 [term 1 LEADER]: Raft consensus shutting down.
I20260925 22:47:37.545128 26936 raft_consensus.cc:2258] T 1220ed2f8c4a4138b6b4091903e8b418 P 1b6eadd191934a76ba315f4c414dd647 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 22:47:37.545730 26935 ts_tablet_manager.cc:1905] T 1220ed2f8c4a4138b6b4091903e8b418 P e299c7a0d9b04d1a8d30de149122fc48: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 22:47:37.546968 26936 ts_tablet_manager.cc:1905] T 1220ed2f8c4a4138b6b4091903e8b418 P 1b6eadd191934a76ba315f4c414dd647: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260925 22:47:37.547801 26420 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.10.198.1:36769 (error 108)
I20260925 22:47:37.548761 11032 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrWXm8X/build/debug/bin/kudu with pid 26632
I20260925 22:47:37.549762 26935 ts_tablet_manager.cc:1918] T 1220ed2f8c4a4138b6b4091903e8b418 P e299c7a0d9b04d1a8d30de149122fc48: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 22:47:37.549888 26935 log.cc:1196] T 1220ed2f8c4a4138b6b4091903e8b418 P e299c7a0d9b04d1a8d30de149122fc48: Deleting WAL directory at /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376259650642-11032-0/minicluster-data/ts-2/wal/wals/1220ed2f8c4a4138b6b4091903e8b418
I20260925 22:47:37.550252 26935 ts_tablet_manager.cc:1939] T 1220ed2f8c4a4138b6b4091903e8b418 P e299c7a0d9b04d1a8d30de149122fc48: Deleting consensus metadata
I20260925 22:47:37.551508 26908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:47:37.551172 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.198.1:36769 (local address 127.0.0.1:42800)
0925 22:47:37.551326 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0925 22:47:37.551417 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.198.1:36769: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
W20260925 22:47:37.551810 26420 catalog_manager.cc:4584] TS 84c93dfd42c94d188c85a636289a4e23 (127.10.198.1:36769): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 1220ed2f8c4a4138b6b4091903e8b418: Network error: Client connection negotiation failed: client connection to 127.10.198.1:36769: connect: Connection refused (error 111)
I20260925 22:47:37.552150 26420 catalog_manager.cc:4841] TS e299c7a0d9b04d1a8d30de149122fc48 (127.10.198.3:43405): tablet 1220ed2f8c4a4138b6b4091903e8b418 (table test-table [id=aef2b90a17464ff58985620109391cb2]) successfully deleted
I20260925 22:47:37.555992 11032 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrWXm8X/build/debug/bin/kudu with pid 26766
I20260925 22:47:37.557456 26908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:47:37.557038 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.198.2:40709 (local address 127.0.0.1:33540)
0925 22:47:37.557249 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0925 22:47:37.557328 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.198.2:40709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98}
I20260925 22:47:37.562371 11032 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrWXm8X/build/debug/bin/kudu with pid 26404
I20260925 22:47:57.569672 11032 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrWXm8X/build/debug/bin/kudu
/tmp/dist-test-taskrWXm8X/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376259650642-11032-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376259650642-11032-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376259650642-11032-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376259650642-11032-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.198.62:41557
--webserver_interface=127.10.198.62
--webserver_port=35045
--keytab_file=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376259650642-11032-0/krb5kdc/kudu_127.10.198.62.keytab
--principal=kudu/127.10.198.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.198.20:41079
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.198.62:41557
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376259650642-11032-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376259650642-11032-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376259650642-11032-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:47:57.691628 26938 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:47:57.692070 26938 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:47:57.692170 26938 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:47:57.696632 26938 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260925 22:47:57.696758 26938 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260925 22:47:57.696789 26938 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260925 22:47:57.696813 26938 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 22:47:57.696857 26938 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:47:57.696925 26938 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:47:57.696964 26938 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 22:47:57.697002 26938 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 22:47:57.702972 26938 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.198.20:41079
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376259650642-11032-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376259650642-11032-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.10.198.62:41557
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376259650642-11032-0/krb5kdc/kudu_127.10.198.62.keytab
--principal=kudu/127.10.198.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.198.62:41557
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376259650642-11032-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.198.62
--webserver_port=35045
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26938
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376259650642-11032-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:09:16 UTC on ccycloud.quasar-afmhyw.root.comops.site
build id 4673379
I20260925 22:47:57.704312 26938 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:47:57.705427 26938 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:47:57.719879 26945 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:47:57.719899 26947 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:47:57.719964 26944 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:47:57.720294 26938 server_base.cc:1034] running on GCE node
I20260925 22:47:57.721371 26938 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:47:57.722815 26938 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:47:57.724010 26938 hybrid_clock.cc:648] HybridClock initialized: now 1790376477723972 us; error 56 us; skew 500 ppm
Sep 25 22:47:57 dist-test-slave-5vdr krb5kdc[26391](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376477, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.198.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:47:57.734724 26938 init.cc:377] Logged in from keytab as kudu/127.10.198.62@KRBTEST.COM (short username kudu)
I20260925 22:47:57.736099 26938 webserver.cc:460] Webserver started at http://127.10.198.62:35045/ using document root <none> and password file <none>
I20260925 22:47:57.736840 26938 fs_manager.cc:359] Metadata directory not provided
I20260925 22:47:57.736982 26938 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:47:57.741696 26938 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260925 22:47:57.744436 26954 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:47:57.745950 26938 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 22:47:57.746146 26938 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376259650642-11032-0/minicluster-data/master-0/data,/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376259650642-11032-0/minicluster-data/master-0/wal
uuid: "3f1861b6350a4355939000f5afa487a5"
format_stamp: "Formatted at 2026-09-25 22:47:36 on dist-test-slave-5vdr"
server_key: "3e784eed9d2d48329bac4e54677457fc"
server_key_iv: "5da66deeb4410b5f5519f8ab958f454f"
server_key_version: "encryptionkey@0"
I20260925 22:47:57.746873 26938 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376259650642-11032-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376259650642-11032-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376259650642-11032-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:47:57.765638 26938 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:47:57.769632 26938 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:47:57.769876 26938 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:47:57.777784 26938 rpc_server.cc:307] RPC server started. Bound to: 127.10.198.62:41557
I20260925 22:47:57.778095 27006 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.198.62:41557 every 8 connection(s)
I20260925 22:47:57.779081 26938 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376259650642-11032-0/minicluster-data/master-0/data/info.pb
I20260925 22:47:57.784112 27007 sys_catalog.cc:263] Verifying existing consensus state
I20260925 22:47:57.786278 27007 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3f1861b6350a4355939000f5afa487a5: Bootstrap starting.
I20260925 22:47:57.788931 11032 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrWXm8X/build/debug/bin/kudu as pid 26938
I20260925 22:47:57.789404 11032 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrWXm8X/build/debug/bin/kudu
/tmp/dist-test-taskrWXm8X/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376259650642-11032-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376259650642-11032-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376259650642-11032-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376259650642-11032-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.198.1:36769
--local_ip_for_outbound_sockets=127.10.198.1
--tserver_master_addrs=127.10.198.62:41557
--webserver_port=34609
--webserver_interface=127.10.198.1
--keytab_file=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376259650642-11032-0/krb5kdc/kudu_127.10.198.1.keytab
--principal=kudu/127.10.198.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.198.20:41079
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376259650642-11032-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376259650642-11032-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376259650642-11032-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 22:47:57.804972 27007 log.cc:824] T 00000000000000000000000000000000 P 3f1861b6350a4355939000f5afa487a5: Log is configured to *not* fsync() on all Append() calls
I20260925 22:47:57.815625 27007 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3f1861b6350a4355939000f5afa487a5: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 22:47:57.816186 27007 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3f1861b6350a4355939000f5afa487a5: Bootstrap complete.
I20260925 22:47:57.821637 27007 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3f1861b6350a4355939000f5afa487a5 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f1861b6350a4355939000f5afa487a5" member_type: VOTER last_known_addr { host: "127.10.198.62" port: 41557 } }
I20260925 22:47:57.822189 27007 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3f1861b6350a4355939000f5afa487a5 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3f1861b6350a4355939000f5afa487a5, State: Initialized, Role: FOLLOWER
I20260925 22:47:57.822774 27007 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3f1861b6350a4355939000f5afa487a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f1861b6350a4355939000f5afa487a5" member_type: VOTER last_known_addr { host: "127.10.198.62" port: 41557 } }
I20260925 22:47:57.822958 27007 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3f1861b6350a4355939000f5afa487a5 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 22:47:57.823089 27007 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3f1861b6350a4355939000f5afa487a5 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 22:47:57.823225 27007 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3f1861b6350a4355939000f5afa487a5 [term 1 FOLLOWER]: Advancing to term 2
I20260925 22:47:57.825721 27007 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3f1861b6350a4355939000f5afa487a5 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f1861b6350a4355939000f5afa487a5" member_type: VOTER last_known_addr { host: "127.10.198.62" port: 41557 } }
I20260925 22:47:57.826189 27007 leader_election.cc:302] T 00000000000000000000000000000000 P 3f1861b6350a4355939000f5afa487a5 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3f1861b6350a4355939000f5afa487a5; no voters:
I20260925 22:47:57.826608 27007 leader_election.cc:288] T 00000000000000000000000000000000 P 3f1861b6350a4355939000f5afa487a5 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260925 22:47:57.826730 27011 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3f1861b6350a4355939000f5afa487a5 [term 2 FOLLOWER]: Leader election won for term 2
I20260925 22:47:57.826977 27011 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3f1861b6350a4355939000f5afa487a5 [term 2 LEADER]: Becoming Leader. State: Replica: 3f1861b6350a4355939000f5afa487a5, State: Running, Role: LEADER
I20260925 22:47:57.827340 27011 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3f1861b6350a4355939000f5afa487a5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f1861b6350a4355939000f5afa487a5" member_type: VOTER last_known_addr { host: "127.10.198.62" port: 41557 } }
I20260925 22:47:57.828191 27007 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3f1861b6350a4355939000f5afa487a5 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 22:47:57.828997 27012 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3f1861b6350a4355939000f5afa487a5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "3f1861b6350a4355939000f5afa487a5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f1861b6350a4355939000f5afa487a5" member_type: VOTER last_known_addr { host: "127.10.198.62" port: 41557 } } }
I20260925 22:47:57.829154 27012 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3f1861b6350a4355939000f5afa487a5 [sys.catalog]: This master's current role is: LEADER
I20260925 22:47:57.829914 27013 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3f1861b6350a4355939000f5afa487a5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3f1861b6350a4355939000f5afa487a5. Latest consensus state: current_term: 2 leader_uuid: "3f1861b6350a4355939000f5afa487a5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f1861b6350a4355939000f5afa487a5" member_type: VOTER last_known_addr { host: "127.10.198.62" port: 41557 } } }
I20260925 22:47:57.830053 27013 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3f1861b6350a4355939000f5afa487a5 [sys.catalog]: This master's current role is: LEADER
I20260925 22:47:57.830412 27018 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 22:47:57.836357 27018 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 22:47:57.837200 27018 catalog_manager.cc:1259] Loaded cluster ID: ff32c441e4cc4360a3fe479cb6485ef5
I20260925 22:47:57.837293 27018 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 22:47:57.840148 27018 catalog_manager.cc:1504] Loading token signing keys...
I20260925 22:47:57.841161 27018 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 3f1861b6350a4355939000f5afa487a5: Loaded TSK: 0
I20260925 22:47:57.854854 27018 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3f1861b6350a4355939000f5afa487a5: Generated new TSK 1
I20260925 22:47:57.855189 27018 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260925 22:47:57.924837 27009 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:47:57.925203 27009 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:47:57.925303 27009 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:47:57.931402 27009 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:47:57.931531 27009 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:47:57.931638 27009 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.198.1
I20260925 22:47:57.938712 27009 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.198.20:41079
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376259650642-11032-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376259650642-11032-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376259650642-11032-0/krb5kdc/kudu_127.10.198.1.keytab
--principal=kudu/127.10.198.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.198.1:36769
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376259650642-11032-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.198.1
--webserver_port=34609
--tserver_master_addrs=127.10.198.62:41557
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27009
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.198.1
--log_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376259650642-11032-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:09:16 UTC on ccycloud.quasar-afmhyw.root.comops.site
build id 4673379
I20260925 22:47:57.940301 27009 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:47:57.941586 27009 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:47:57.954016 27035 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:47:57.954115 27037 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:47:57.954115 27034 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:47:57.955027 27009 server_base.cc:1034] running on GCE node
I20260925 22:47:57.955549 27009 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:47:57.956496 27009 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:47:57.957751 27009 hybrid_clock.cc:648] HybridClock initialized: now 1790376477957713 us; error 64 us; skew 500 ppm
Sep 25 22:47:57 dist-test-slave-5vdr krb5kdc[26391](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376477, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.198.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:47:57.961124 27009 init.cc:377] Logged in from keytab as kudu/127.10.198.1@KRBTEST.COM (short username kudu)
I20260925 22:47:57.962399 27009 webserver.cc:460] Webserver started at http://127.10.198.1:34609/ using document root <none> and password file <none>
I20260925 22:47:57.963078 27009 fs_manager.cc:359] Metadata directory not provided
I20260925 22:47:57.963168 27009 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:47:57.968315 27009 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260925 22:47:57.971145 27044 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:47:57.972790 27009 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 22:47:57.973053 27009 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376259650642-11032-0/minicluster-data/ts-0/data,/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376259650642-11032-0/minicluster-data/ts-0/wal
uuid: "84c93dfd42c94d188c85a636289a4e23"
format_stamp: "Formatted at 2026-09-25 22:47:36 on dist-test-slave-5vdr"
server_key: "e0c3fa3a013ecdfc35c9ebe8f8f04413"
server_key_iv: "c47103792e3bc31abe6ea5c9c990e8e2"
server_key_version: "encryptionkey@0"
I20260925 22:47:57.973791 27009 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376259650642-11032-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376259650642-11032-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376259650642-11032-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:47:57.994050 27009 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:47:57.997947 27009 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:47:57.998172 27009 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:47:57.998876 27009 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 22:47:58.000250 27051 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260925 22:47:58.004899 27009 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260925 22:47:58.005000 27009 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.005s user 0.000s sys 0.000s
I20260925 22:47:58.005060 27009 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260925 22:47:58.007025 27009 ts_tablet_manager.cc:610] Registered 1 tablets
I20260925 22:47:58.007092 27009 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260925 22:47:58.007503 27051 tablet_bootstrap.cc:492] T 1220ed2f8c4a4138b6b4091903e8b418 P 84c93dfd42c94d188c85a636289a4e23: Bootstrap starting.
I20260925 22:47:58.018559 27009 rpc_server.cc:307] RPC server started. Bound to: 127.10.198.1:36769
I20260925 22:47:58.018594 27158 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.198.1:36769 every 8 connection(s)
I20260925 22:47:58.019711 27009 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376259650642-11032-0/minicluster-data/ts-0/data/info.pb
I20260925 22:47:58.027743 27051 log.cc:824] T 1220ed2f8c4a4138b6b4091903e8b418 P 84c93dfd42c94d188c85a636289a4e23: Log is configured to *not* fsync() on all Append() calls
I20260925 22:47:58.027904 11032 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrWXm8X/build/debug/bin/kudu as pid 27009
I20260925 22:47:58.028443 11032 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrWXm8X/build/debug/bin/kudu
/tmp/dist-test-taskrWXm8X/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376259650642-11032-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376259650642-11032-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376259650642-11032-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376259650642-11032-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.198.2:40709
--local_ip_for_outbound_sockets=127.10.198.2
--tserver_master_addrs=127.10.198.62:41557
--webserver_port=40329
--webserver_interface=127.10.198.2
--keytab_file=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376259650642-11032-0/krb5kdc/kudu_127.10.198.2.keytab
--principal=kudu/127.10.198.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.198.20:41079
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376259650642-11032-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376259650642-11032-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376259650642-11032-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 25 22:47:58 dist-test-slave-5vdr krb5kdc[26391](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376477, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.198.1@KRBTEST.COM for kudu/127.10.198.62@KRBTEST.COM
I20260925 22:47:58.039343 27051 tablet_bootstrap.cc:492] T 1220ed2f8c4a4138b6b4091903e8b418 P 84c93dfd42c94d188c85a636289a4e23: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 22:47:58.039949 27051 tablet_bootstrap.cc:492] T 1220ed2f8c4a4138b6b4091903e8b418 P 84c93dfd42c94d188c85a636289a4e23: Bootstrap complete.
I20260925 22:47:58.041030 27162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:47:58.022434 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.198.62:41557 (local address 127.10.198.1:56489)
0925 22:47:58.023100 (+ 666us) negotiation.cc:107] Waiting for socket to connect
0925 22:47:58.023162 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0925 22:47:58.024527 (+ 1365us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:47:58.026368 (+ 1841us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:47:58.026377 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:47:58.026820 (+ 443us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:47:58.027379 (+ 559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:47:58.027398 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:47:58.029569 (+ 2171us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:47:58.029576 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:47:58.030925 (+ 1349us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:47:58.030940 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:47:58.031836 (+ 896us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:47:58.031868 (+ 32us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:47:58.032079 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:47:58.032776 (+ 697us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:47:58.032795 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:47:58.034660 (+ 1865us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:47:58.037326 (+ 2666us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:47:58.037333 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:47:58.037348 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:47:58.037662 (+ 314us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:47:58.038081 (+ 419us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:47:58.038085 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:47:58.038087 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:47:58.038251 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:47:58.039027 (+ 776us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:47:58.039034 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:47:58.039309 (+ 275us) client_negotiation.cc:769] Sending connection context
0925 22:47:58.039517 (+ 208us) client_negotiation.cc:240] Negotiation successful
0925 22:47:58.039924 (+ 407us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":440,"thread_start_us":165,"threads_started":1}
I20260925 22:47:58.041602 27051 ts_tablet_manager.cc:1397] T 1220ed2f8c4a4138b6b4091903e8b418 P 84c93dfd42c94d188c85a636289a4e23: Time spent bootstrapping tablet: real 0.034s user 0.017s sys 0.015s
I20260925 22:47:58.041638 27161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:47:58.022437 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.198.1:56489 (local address 127.10.198.62:41557)
0925 22:47:58.022977 (+ 540us) server_negotiation.cc:206] Beginning negotiation
0925 22:47:58.023000 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0925 22:47:58.023326 (+ 326us) server_negotiation.cc:407] Connection header received
0925 22:47:58.025188 (+ 1862us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:47:58.025213 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:47:58.025796 (+ 583us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:47:58.026257 (+ 461us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:47:58.027636 (+ 1379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:47:58.029307 (+ 1671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:47:58.031118 (+ 1811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:47:58.031657 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:47:58.031907 (+ 250us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:47:58.034943 (+ 3036us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:47:58.034949 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:47:58.034963 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:47:58.035000 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:47:58.037140 (+ 2140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:47:58.037815 (+ 675us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:47:58.037821 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:47:58.037826 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:47:58.037967 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:47:58.038360 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:47:58.038364 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:47:58.038365 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:47:58.038897 (+ 532us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:47:58.039058 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0925 22:47:58.040159 (+ 1101us) server_negotiation.cc:299] Negotiation successful
0925 22:47:58.040488 (+ 329us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":362,"thread_start_us":153,"threads_started":1}
I20260925 22:47:58.044397 27159 heartbeater.cc:347] Connected to a master server at 127.10.198.62:41557
I20260925 22:47:58.044716 27159 heartbeater.cc:464] Registering TS with master...
I20260925 22:47:58.045284 27159 heartbeater.cc:511] Master 127.10.198.62:41557 requested a full tablet report, sending...
I20260925 22:47:58.044992 27051 raft_consensus.cc:348] T 1220ed2f8c4a4138b6b4091903e8b418 P 84c93dfd42c94d188c85a636289a4e23 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84c93dfd42c94d188c85a636289a4e23" member_type: VOTER last_known_addr { host: "127.10.198.1" port: 36769 } } peers { permanent_uuid: "e299c7a0d9b04d1a8d30de149122fc48" member_type: VOTER last_known_addr { host: "127.10.198.3" port: 43405 } } peers { permanent_uuid: "1b6eadd191934a76ba315f4c414dd647" member_type: VOTER last_known_addr { host: "127.10.198.2" port: 40709 } }
I20260925 22:47:58.045593 27051 raft_consensus.cc:729] T 1220ed2f8c4a4138b6b4091903e8b418 P 84c93dfd42c94d188c85a636289a4e23 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 84c93dfd42c94d188c85a636289a4e23, State: Initialized, Role: FOLLOWER
I20260925 22:47:58.046172 27051 consensus_queue.cc:260] T 1220ed2f8c4a4138b6b4091903e8b418 P 84c93dfd42c94d188c85a636289a4e23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84c93dfd42c94d188c85a636289a4e23" member_type: VOTER last_known_addr { host: "127.10.198.1" port: 36769 } } peers { permanent_uuid: "e299c7a0d9b04d1a8d30de149122fc48" member_type: VOTER last_known_addr { host: "127.10.198.3" port: 43405 } } peers { permanent_uuid: "1b6eadd191934a76ba315f4c414dd647" member_type: VOTER last_known_addr { host: "127.10.198.2" port: 40709 } }
I20260925 22:47:58.047376 27051 ts_tablet_manager.cc:1428] T 1220ed2f8c4a4138b6b4091903e8b418 P 84c93dfd42c94d188c85a636289a4e23: Time spent starting tablet: real 0.006s user 0.006s sys 0.001s
I20260925 22:47:58.049088 26971 ts_manager.cc:194] Registered new tserver with Master: 84c93dfd42c94d188c85a636289a4e23 (127.10.198.1:36769)
I20260925 22:47:58.053346 26971 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.198.1@KRBTEST.COM'} at 127.10.198.1:56489
I20260925 22:47:58.056365 27159 heartbeater.cc:503] Master 127.10.198.62:41557 was elected leader, sending a full tablet report...
I20260925 22:47:58.060479 27167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:47:58.052994 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59414 (local address 127.10.198.1:36769)
0925 22:47:58.054764 (+ 1770us) server_negotiation.cc:206] Beginning negotiation
0925 22:47:58.054779 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0925 22:47:58.054810 (+ 31us) server_negotiation.cc:407] Connection header received
0925 22:47:58.054896 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:47:58.054902 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:47:58.055358 (+ 456us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:47:58.055587 (+ 229us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:47:58.056038 (+ 451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:47:58.057147 (+ 1109us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:47:58.058998 (+ 1851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:47:58.059599 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:47:58.059704 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:47:58.059906 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0925 22:47:58.060251 (+ 345us) server_negotiation.cc:299] Negotiation successful
0925 22:47:58.060350 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1634,"thread_start_us":122,"threads_started":1}
I20260925 22:47:58.060560 27166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:47:58.052871 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.198.1:36769 (local address 127.0.0.1:59414)
0925 22:47:58.054136 (+ 1265us) negotiation.cc:107] Waiting for socket to connect
0925 22:47:58.054204 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0925 22:47:58.054379 (+ 175us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:47:58.055623 (+ 1244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:47:58.055631 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:47:58.055664 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:47:58.055828 (+ 164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:47:58.055865 (+ 37us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:47:58.057334 (+ 1469us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:47:58.057338 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:47:58.058814 (+ 1476us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:47:58.058834 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:47:58.059774 (+ 940us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:47:58.059779 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:47:58.060001 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:47:58.060058 (+ 57us) client_negotiation.cc:769] Sending connection context
0925 22:47:58.060210 (+ 152us) client_negotiation.cc:240] Negotiation successful
0925 22:47:58.060368 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1037,"thread_start_us":170,"threads_started":1}
I20260925 22:47:58.061856 27093 tablet_service.cc:1514] Processing DeleteTablet for tablet 1220ed2f8c4a4138b6b4091903e8b418 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 22:47:37 UTC) from {username='kudu', principal='kudu/127.10.198.62@KRBTEST.COM'} at 127.0.0.1:59414
I20260925 22:47:58.063053 27168 tablet_replica.cc:330] stopping tablet replica
I20260925 22:47:58.063432 27168 raft_consensus.cc:2229] T 1220ed2f8c4a4138b6b4091903e8b418 P 84c93dfd42c94d188c85a636289a4e23 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 22:47:58.063730 27168 raft_consensus.cc:2258] T 1220ed2f8c4a4138b6b4091903e8b418 P 84c93dfd42c94d188c85a636289a4e23 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 22:47:58.064910 27168 ts_tablet_manager.cc:1905] T 1220ed2f8c4a4138b6b4091903e8b418 P 84c93dfd42c94d188c85a636289a4e23: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 22:47:58.069286 27168 ts_tablet_manager.cc:1918] T 1220ed2f8c4a4138b6b4091903e8b418 P 84c93dfd42c94d188c85a636289a4e23: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 22:47:58.069409 27168 log.cc:1196] T 1220ed2f8c4a4138b6b4091903e8b418 P 84c93dfd42c94d188c85a636289a4e23: Deleting WAL directory at /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376259650642-11032-0/minicluster-data/ts-0/wal/wals/1220ed2f8c4a4138b6b4091903e8b418
I20260925 22:47:58.069903 27168 ts_tablet_manager.cc:1939] T 1220ed2f8c4a4138b6b4091903e8b418 P 84c93dfd42c94d188c85a636289a4e23: Deleting consensus metadata
I20260925 22:47:58.072156 26955 catalog_manager.cc:4841] TS 84c93dfd42c94d188c85a636289a4e23 (127.10.198.1:36769): tablet 1220ed2f8c4a4138b6b4091903e8b418 (table test-table [id=aef2b90a17464ff58985620109391cb2]) successfully deleted
W20260925 22:47:58.161641 27164 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:47:58.162042 27164 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:47:58.162129 27164 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:47:58.166692 27164 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:47:58.166834 27164 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:47:58.166950 27164 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.198.2
I20260925 22:47:58.173087 27164 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.198.20:41079
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376259650642-11032-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376259650642-11032-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376259650642-11032-0/krb5kdc/kudu_127.10.198.2.keytab
--principal=kudu/127.10.198.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.198.2:40709
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376259650642-11032-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.198.2
--webserver_port=40329
--tserver_master_addrs=127.10.198.62:41557
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27164
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.198.2
--log_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376259650642-11032-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:09:16 UTC on ccycloud.quasar-afmhyw.root.comops.site
build id 4673379
I20260925 22:47:58.174373 27164 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:47:58.175594 27164 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:47:58.186983 27173 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:47:58.186978 27174 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:47:58.186992 27176 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:47:58.187458 27164 server_base.cc:1034] running on GCE node
I20260925 22:47:58.188005 27164 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:47:58.188778 27164 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:47:58.189966 27164 hybrid_clock.cc:648] HybridClock initialized: now 1790376478189953 us; error 39 us; skew 500 ppm
Sep 25 22:47:58 dist-test-slave-5vdr krb5kdc[26391](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376478, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.198.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:47:58.193217 27164 init.cc:377] Logged in from keytab as kudu/127.10.198.2@KRBTEST.COM (short username kudu)
I20260925 22:47:58.194378 27164 webserver.cc:460] Webserver started at http://127.10.198.2:40329/ using document root <none> and password file <none>
I20260925 22:47:58.194964 27164 fs_manager.cc:359] Metadata directory not provided
I20260925 22:47:58.195048 27164 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:47:58.199653 27164 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260925 22:47:58.202193 27183 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:47:58.203825 27164 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260925 22:47:58.204089 27164 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376259650642-11032-0/minicluster-data/ts-1/data,/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376259650642-11032-0/minicluster-data/ts-1/wal
uuid: "1b6eadd191934a76ba315f4c414dd647"
format_stamp: "Formatted at 2026-09-25 22:47:36 on dist-test-slave-5vdr"
server_key: "5f931a5901e7de1154c19256f2a62735"
server_key_iv: "38c9689b483c352210f94e845e378f22"
server_key_version: "encryptionkey@0"
I20260925 22:47:58.204728 27164 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376259650642-11032-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376259650642-11032-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376259650642-11032-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:47:58.216130 27164 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:47:58.219790 27164 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:47:58.220094 27164 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:47:58.220780 27164 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 22:47:58.222030 27190 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
W20260925 22:47:58.226375 27190 ts_tablet_manager.cc:1852] T 1220ed2f8c4a4138b6b4091903e8b418 P 1b6eadd191934a76ba315f4c414dd647: Tablet Manager startup: Rolling forward tablet deletion of type TABLET_DATA_DELETED
I20260925 22:47:58.226874 27190 ts_tablet_manager.cc:1905] T 1220ed2f8c4a4138b6b4091903e8b418 P 1b6eadd191934a76ba315f4c414dd647: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 22:47:58.231784 27190 ts_tablet_manager.cc:1918] T 1220ed2f8c4a4138b6b4091903e8b418 P 1b6eadd191934a76ba315f4c414dd647: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 22:47:58.231997 27190 log.cc:1196] T 1220ed2f8c4a4138b6b4091903e8b418 P 1b6eadd191934a76ba315f4c414dd647: Deleting WAL directory at /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376259650642-11032-0/minicluster-data/ts-1/wal/wals/1220ed2f8c4a4138b6b4091903e8b418
I20260925 22:47:58.232405 27190 ts_tablet_manager.cc:1939] T 1220ed2f8c4a4138b6b4091903e8b418 P 1b6eadd191934a76ba315f4c414dd647: Deleting consensus metadata
I20260925 22:47:58.232793 27164 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 0 live tablets)
I20260925 22:47:58.232867 27164 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.011s user 0.001s sys 0.000s
I20260925 22:47:58.232932 27164 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:47:58.232955 27164 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 22:47:58.244915 27164 rpc_server.cc:307] RPC server started. Bound to: 127.10.198.2:40709
I20260925 22:47:58.245031 27297 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.198.2:40709 every 8 connection(s)
I20260925 22:47:58.246060 27164 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376259650642-11032-0/minicluster-data/ts-1/data/info.pb
I20260925 22:47:58.247548 11032 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrWXm8X/build/debug/bin/kudu as pid 27164
I20260925 22:47:58.248178 11032 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrWXm8X/build/debug/bin/kudu
/tmp/dist-test-taskrWXm8X/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376259650642-11032-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376259650642-11032-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376259650642-11032-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376259650642-11032-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.198.3:43405
--local_ip_for_outbound_sockets=127.10.198.3
--tserver_master_addrs=127.10.198.62:41557
--webserver_port=43119
--webserver_interface=127.10.198.3
--keytab_file=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376259650642-11032-0/krb5kdc/kudu_127.10.198.3.keytab
--principal=kudu/127.10.198.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.198.20:41079
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376259650642-11032-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376259650642-11032-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376259650642-11032-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 25 22:47:58 dist-test-slave-5vdr krb5kdc[26391](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376478, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.198.2@KRBTEST.COM for kudu/127.10.198.62@KRBTEST.COM
I20260925 22:47:58.264009 27161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:47:58.248321 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.198.2:37363 (local address 127.10.198.62:41557)
0925 22:47:58.248540 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0925 22:47:58.248544 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:47:58.250001 (+ 1457us) server_negotiation.cc:407] Connection header received
0925 22:47:58.251567 (+ 1566us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:47:58.251573 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:47:58.251664 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:47:58.251796 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:47:58.253215 (+ 1419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:47:58.254179 (+ 964us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:47:58.255649 (+ 1470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:47:58.256146 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:47:58.256311 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:47:58.259387 (+ 3076us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:47:58.259395 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:47:58.259399 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:47:58.259443 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:47:58.261385 (+ 1942us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:47:58.262061 (+ 676us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:47:58.262064 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:47:58.262066 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:47:58.262172 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:47:58.262551 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:47:58.262554 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:47:58.262556 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:47:58.262797 (+ 241us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:47:58.262931 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0925 22:47:58.263619 (+ 688us) server_negotiation.cc:299] Negotiation successful
0925 22:47:58.263785 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":132}
I20260925 22:47:58.265216 27300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:47:58.248993 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.198.62:41557 (local address 127.10.198.2:37363)
0925 22:47:58.249748 (+ 755us) negotiation.cc:107] Waiting for socket to connect
0925 22:47:58.249835 (+ 87us) client_negotiation.cc:174] Beginning negotiation
0925 22:47:58.251218 (+ 1383us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:47:58.252049 (+ 831us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:47:58.252058 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:47:58.252458 (+ 400us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:47:58.253017 (+ 559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:47:58.253033 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:47:58.254397 (+ 1364us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:47:58.254401 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:47:58.255481 (+ 1080us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:47:58.255494 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:47:58.256338 (+ 844us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:47:58.256342 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:47:58.256524 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:47:58.257297 (+ 773us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:47:58.257317 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:47:58.259116 (+ 1799us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:47:58.261559 (+ 2443us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:47:58.261567 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:47:58.261586 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:47:58.261937 (+ 351us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:47:58.262279 (+ 342us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:47:58.262281 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:47:58.262283 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:47:58.262451 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:47:58.262990 (+ 539us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:47:58.262999 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:47:58.263339 (+ 340us) client_negotiation.cc:769] Sending connection context
0925 22:47:58.263556 (+ 217us) client_negotiation.cc:240] Negotiation successful
0925 22:47:58.263982 (+ 426us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":455,"thread_start_us":202,"threads_started":1}
I20260925 22:47:58.266919 27298 heartbeater.cc:347] Connected to a master server at 127.10.198.62:41557
I20260925 22:47:58.267261 27298 heartbeater.cc:464] Registering TS with master...
I20260925 22:47:58.267968 27298 heartbeater.cc:511] Master 127.10.198.62:41557 requested a full tablet report, sending...
I20260925 22:47:58.269357 26971 ts_manager.cc:194] Registered new tserver with Master: 1b6eadd191934a76ba315f4c414dd647 (127.10.198.2:40709)
I20260925 22:47:58.270012 26971 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.198.2@KRBTEST.COM'} at 127.10.198.2:37363
W20260925 22:47:58.375074 27301 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:47:58.375473 27301 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:47:58.375561 27301 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:47:58.380244 27301 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:47:58.380366 27301 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:47:58.380467 27301 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.198.3
I20260925 22:47:58.387313 27301 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.198.20:41079
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376259650642-11032-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376259650642-11032-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376259650642-11032-0/krb5kdc/kudu_127.10.198.3.keytab
--principal=kudu/127.10.198.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.198.3:43405
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376259650642-11032-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.198.3
--webserver_port=43119
--tserver_master_addrs=127.10.198.62:41557
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27301
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.198.3
--log_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376259650642-11032-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:09:16 UTC on ccycloud.quasar-afmhyw.root.comops.site
build id 4673379
I20260925 22:47:58.388720 27301 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:47:58.390007 27301 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:47:58.400877 27306 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:47:58.401036 27301 server_base.cc:1034] running on GCE node
W20260925 22:47:58.400892 27307 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:47:58.400892 27309 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:47:58.401701 27301 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:47:58.402464 27301 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:47:58.403705 27301 hybrid_clock.cc:648] HybridClock initialized: now 1790376478403661 us; error 59 us; skew 500 ppm
Sep 25 22:47:58 dist-test-slave-5vdr krb5kdc[26391](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376478, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.198.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:47:58.407230 27301 init.cc:377] Logged in from keytab as kudu/127.10.198.3@KRBTEST.COM (short username kudu)
I20260925 22:47:58.408545 27301 webserver.cc:460] Webserver started at http://127.10.198.3:43119/ using document root <none> and password file <none>
I20260925 22:47:58.409202 27301 fs_manager.cc:359] Metadata directory not provided
I20260925 22:47:58.409354 27301 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:47:58.414209 27301 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260925 22:47:58.416872 27316 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:47:58.418504 27301 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 22:47:58.418675 27301 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376259650642-11032-0/minicluster-data/ts-2/data,/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376259650642-11032-0/minicluster-data/ts-2/wal
uuid: "e299c7a0d9b04d1a8d30de149122fc48"
format_stamp: "Formatted at 2026-09-25 22:47:37 on dist-test-slave-5vdr"
server_key: "5e2b92cea1a43418998f5092402082b4"
server_key_iv: "8e5ef74a1b33c58e90f390b421cc177c"
server_key_version: "encryptionkey@0"
I20260925 22:47:58.419399 27301 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376259650642-11032-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376259650642-11032-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376259650642-11032-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:47:58.438381 27301 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:47:58.442453 27301 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:47:58.442729 27301 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:47:58.443450 27301 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 22:47:58.444552 27301 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:47:58.444620 27301 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:47:58.444702 27301 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:47:58.444742 27301 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 22:47:58.456259 27301 rpc_server.cc:307] RPC server started. Bound to: 127.10.198.3:43405
I20260925 22:47:58.456313 27429 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.198.3:43405 every 8 connection(s)
I20260925 22:47:58.457463 27301 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376259650642-11032-0/minicluster-data/ts-2/data/info.pb
I20260925 22:47:58.467592 11032 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrWXm8X/build/debug/bin/kudu as pid 27301
Sep 25 22:47:58 dist-test-slave-5vdr krb5kdc[26391](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376478, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.198.3@KRBTEST.COM for kudu/127.10.198.62@KRBTEST.COM
I20260925 22:47:58.474035 27161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:47:58.459688 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.198.3:45777 (local address 127.10.198.62:41557)
0925 22:47:58.459911 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0925 22:47:58.459916 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:47:58.460994 (+ 1078us) server_negotiation.cc:407] Connection header received
0925 22:47:58.462468 (+ 1474us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:47:58.462473 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:47:58.462542 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:47:58.462814 (+ 272us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:47:58.464513 (+ 1699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:47:58.465058 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:47:58.466419 (+ 1361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:47:58.466664 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:47:58.466794 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:47:58.470075 (+ 3281us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:47:58.470078 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:47:58.470080 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:47:58.470105 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:47:58.471692 (+ 1587us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:47:58.472368 (+ 676us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:47:58.472372 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:47:58.472373 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:47:58.472452 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:47:58.472842 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:47:58.472845 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:47:58.472849 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:47:58.473037 (+ 188us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:47:58.473148 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0925 22:47:58.473722 (+ 574us) server_negotiation.cc:299] Negotiation successful
0925 22:47:58.473867 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260925 22:47:58.475164 27432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:47:58.460255 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.198.62:41557 (local address 127.10.198.3:45777)
0925 22:47:58.460801 (+ 546us) negotiation.cc:107] Waiting for socket to connect
0925 22:47:58.460854 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0925 22:47:58.462178 (+ 1324us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:47:58.463031 (+ 853us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:47:58.463040 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:47:58.463491 (+ 451us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:47:58.464295 (+ 804us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:47:58.464326 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:47:58.465237 (+ 911us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:47:58.465241 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:47:58.466244 (+ 1003us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:47:58.466258 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:47:58.466855 (+ 597us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:47:58.466859 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:47:58.467058 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:47:58.467989 (+ 931us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:47:58.468009 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:47:58.469877 (+ 1868us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:47:58.471880 (+ 2003us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:47:58.471888 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:47:58.471910 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:47:58.472234 (+ 324us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:47:58.472548 (+ 314us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:47:58.472552 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:47:58.472554 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:47:58.472730 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:47:58.473146 (+ 416us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:47:58.473155 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:47:58.473461 (+ 306us) client_negotiation.cc:769] Sending connection context
0925 22:47:58.473666 (+ 205us) client_negotiation.cc:240] Negotiation successful
0925 22:47:58.474038 (+ 372us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":330,"thread_start_us":141,"threads_started":1}
I20260925 22:47:58.476572 27430 heartbeater.cc:347] Connected to a master server at 127.10.198.62:41557
I20260925 22:47:58.476874 27430 heartbeater.cc:464] Registering TS with master...
I20260925 22:47:58.477439 27430 heartbeater.cc:511] Master 127.10.198.62:41557 requested a full tablet report, sending...
I20260925 22:47:58.478302 27434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:47:58.468493 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39360 (local address 127.10.198.62:41557)
0925 22:47:58.468799 (+ 306us) server_negotiation.cc:206] Beginning negotiation
0925 22:47:58.468806 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 22:47:58.468918 (+ 112us) server_negotiation.cc:407] Connection header received
0925 22:47:58.469094 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:47:58.469097 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:47:58.469158 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:47:58.469263 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:47:58.470090 (+ 827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:47:58.470798 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:47:58.471757 (+ 959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:47:58.472067 (+ 310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:47:58.472195 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:47:58.473990 (+ 1795us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:47:58.473995 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:47:58.473997 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:47:58.474022 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:47:58.476267 (+ 2245us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:47:58.476806 (+ 539us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:47:58.476816 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:47:58.476817 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:47:58.476905 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:47:58.477237 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:47:58.477240 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:47:58.477242 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:47:58.477495 (+ 253us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:47:58.477751 (+ 256us) server_negotiation.cc:1035] Waiting for connection context
0925 22:47:58.477931 (+ 180us) server_negotiation.cc:299] Negotiation successful
0925 22:47:58.478051 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":241,"thread_start_us":135,"threads_started":1}
I20260925 22:47:58.478794 26971 ts_manager.cc:194] Registered new tserver with Master: e299c7a0d9b04d1a8d30de149122fc48 (127.10.198.3:43405)
I20260925 22:47:58.479593 26971 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.198.3@KRBTEST.COM'} at 127.10.198.3:45777
I20260925 22:47:58.480314 11032 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 22:47:58.484858 27434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:47:58.481006 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39374 (local address 127.10.198.62:41557)
0925 22:47:58.481151 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0925 22:47:58.481156 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:47:58.481379 (+ 223us) server_negotiation.cc:407] Connection header received
0925 22:47:58.481544 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:47:58.481546 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:47:58.481638 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:47:58.481746 (+ 108us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 22:47:58.482024 (+ 278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:47:58.482552 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:47:58.483539 (+ 987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:47:58.483823 (+ 284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:47:58.484003 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:47:58.484287 (+ 284us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 22:47:58.484430 (+ 143us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0925 22:47:58.484658 (+ 228us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:39374: token expired
Metrics: {"server-negotiator.queue_time_us":59}
W20260925 22:47:58.484956 27434 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:39374: token expired
W20260925 22:47:58.485325 26900 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.10.198.62:41557)
I20260925 22:47:58.493340 27434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:47:58.485723 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39378 (local address 127.10.198.62:41557)
0925 22:47:58.485855 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0925 22:47:58.485859 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:47:58.485872 (+ 13us) server_negotiation.cc:407] Connection header received
0925 22:47:58.485963 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:47:58.485965 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:47:58.486004 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:47:58.486070 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:47:58.486695 (+ 625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:47:58.487234 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:47:58.488198 (+ 964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:47:58.488417 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:47:58.488505 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:47:58.489879 (+ 1374us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:47:58.489882 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:47:58.489885 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:47:58.489914 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:47:58.491309 (+ 1395us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:47:58.491817 (+ 508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:47:58.491820 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:47:58.491821 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:47:58.491993 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:47:58.492296 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:47:58.492300 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:47:58.492301 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:47:58.492580 (+ 279us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:47:58.492798 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0925 22:47:58.493070 (+ 272us) server_negotiation.cc:299] Negotiation successful
0925 22:47:58.493195 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260925 22:47:58.496817 26970 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39378:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 22:47:58.500442 26970 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 22:47:58.509869 27093 tablet_service.cc:1467] Processing CreateTablet for tablet 44c9a15c2ed944aebff7390622aa8768 (DEFAULT_TABLE table=test-table [id=4a30dcb06cea4fdfba925f7dd695e630]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 22:47:58.510591 27093 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 44c9a15c2ed944aebff7390622aa8768. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:47:58.513520 27051 tablet_bootstrap.cc:492] T 44c9a15c2ed944aebff7390622aa8768 P 84c93dfd42c94d188c85a636289a4e23: Bootstrap starting.
I20260925 22:47:58.514417 27166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:47:58.508512 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.198.3:43405 (local address 127.0.0.1:33992)
0925 22:47:58.508874 (+ 362us) negotiation.cc:107] Waiting for socket to connect
0925 22:47:58.508892 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0925 22:47:58.508986 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:47:58.509847 (+ 861us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:47:58.509850 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:47:58.509873 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:47:58.510006 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:47:58.510013 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:47:58.511510 (+ 1497us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:47:58.511514 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:47:58.512749 (+ 1235us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:47:58.512763 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:47:58.513997 (+ 1234us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:47:58.514001 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:47:58.514093 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:47:58.514116 (+ 23us) client_negotiation.cc:769] Sending connection context
0925 22:47:58.514186 (+ 70us) client_negotiation.cc:240] Negotiation successful
0925 22:47:58.514278 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":249}
I20260925 22:47:58.514470 27439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:47:58.508665 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33992 (local address 127.10.198.3:43405)
0925 22:47:58.509301 (+ 636us) server_negotiation.cc:206] Beginning negotiation
0925 22:47:58.509316 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0925 22:47:58.509340 (+ 24us) server_negotiation.cc:407] Connection header received
0925 22:47:58.509428 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:47:58.509434 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:47:58.509674 (+ 240us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:47:58.509908 (+ 234us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:47:58.510104 (+ 196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:47:58.511198 (+ 1094us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:47:58.512902 (+ 1704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:47:58.513839 (+ 937us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:47:58.513991 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:47:58.514163 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0925 22:47:58.514233 (+ 70us) server_negotiation.cc:299] Negotiation successful
0925 22:47:58.514330 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":518,"thread_start_us":153,"threads_started":1}
I20260925 22:47:58.514633 27051 tablet_bootstrap.cc:654] T 44c9a15c2ed944aebff7390622aa8768 P 84c93dfd42c94d188c85a636289a4e23: Neither blocks nor log segments found. Creating new log.
I20260925 22:47:58.516098 27051 tablet_bootstrap.cc:492] T 44c9a15c2ed944aebff7390622aa8768 P 84c93dfd42c94d188c85a636289a4e23: No bootstrap required, opened a new log
I20260925 22:47:58.516240 27051 ts_tablet_manager.cc:1397] T 44c9a15c2ed944aebff7390622aa8768 P 84c93dfd42c94d188c85a636289a4e23: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.001s
I20260925 22:47:58.516760 27051 raft_consensus.cc:348] T 44c9a15c2ed944aebff7390622aa8768 P 84c93dfd42c94d188c85a636289a4e23 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e299c7a0d9b04d1a8d30de149122fc48" member_type: VOTER last_known_addr { host: "127.10.198.3" port: 43405 } } peers { permanent_uuid: "1b6eadd191934a76ba315f4c414dd647" member_type: VOTER last_known_addr { host: "127.10.198.2" port: 40709 } } peers { permanent_uuid: "84c93dfd42c94d188c85a636289a4e23" member_type: VOTER last_known_addr { host: "127.10.198.1" port: 36769 } }
I20260925 22:47:58.516932 27051 raft_consensus.cc:374] T 44c9a15c2ed944aebff7390622aa8768 P 84c93dfd42c94d188c85a636289a4e23 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:47:58.516978 27051 raft_consensus.cc:729] T 44c9a15c2ed944aebff7390622aa8768 P 84c93dfd42c94d188c85a636289a4e23 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 84c93dfd42c94d188c85a636289a4e23, State: Initialized, Role: FOLLOWER
I20260925 22:47:58.517181 27051 consensus_queue.cc:260] T 44c9a15c2ed944aebff7390622aa8768 P 84c93dfd42c94d188c85a636289a4e23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e299c7a0d9b04d1a8d30de149122fc48" member_type: VOTER last_known_addr { host: "127.10.198.3" port: 43405 } } peers { permanent_uuid: "1b6eadd191934a76ba315f4c414dd647" member_type: VOTER last_known_addr { host: "127.10.198.2" port: 40709 } } peers { permanent_uuid: "84c93dfd42c94d188c85a636289a4e23" member_type: VOTER last_known_addr { host: "127.10.198.1" port: 36769 } }
I20260925 22:47:58.517519 27440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:47:58.508843 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.198.2:40709 (local address 127.0.0.1:54768)
0925 22:47:58.509337 (+ 494us) negotiation.cc:107] Waiting for socket to connect
0925 22:47:58.509355 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0925 22:47:58.509452 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:47:58.510791 (+ 1339us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:47:58.510794 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:47:58.510816 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:47:58.511171 (+ 355us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:47:58.511181 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:47:58.512775 (+ 1594us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:47:58.512778 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:47:58.515446 (+ 2668us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:47:58.515463 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:47:58.516686 (+ 1223us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:47:58.516691 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:47:58.517081 (+ 390us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:47:58.517122 (+ 41us) client_negotiation.cc:769] Sending connection context
0925 22:47:58.517229 (+ 107us) client_negotiation.cc:240] Negotiation successful
0925 22:47:58.517336 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":408,"thread_start_us":91,"threads_started":1}
I20260925 22:47:58.517817 27441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:47:58.509172 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54768 (local address 127.10.198.2:40709)
0925 22:47:58.509717 (+ 545us) server_negotiation.cc:206] Beginning negotiation
0925 22:47:58.509727 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 22:47:58.509748 (+ 21us) server_negotiation.cc:407] Connection header received
0925 22:47:58.509852 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:47:58.509858 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:47:58.510478 (+ 620us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:47:58.510968 (+ 490us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:47:58.511318 (+ 350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:47:58.512608 (+ 1290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:47:58.515633 (+ 3025us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:47:58.516523 (+ 890us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:47:58.516648 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:47:58.516812 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0925 22:47:58.517284 (+ 472us) server_negotiation.cc:299] Negotiation successful
0925 22:47:58.517500 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":447,"thread_start_us":164,"threads_started":1}
I20260925 22:47:58.518903 27051 ts_tablet_manager.cc:1428] T 44c9a15c2ed944aebff7390622aa8768 P 84c93dfd42c94d188c85a636289a4e23: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20260925 22:47:58.518651 27364 tablet_service.cc:1467] Processing CreateTablet for tablet 44c9a15c2ed944aebff7390622aa8768 (DEFAULT_TABLE table=test-table [id=4a30dcb06cea4fdfba925f7dd695e630]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 22:47:58.519593 27232 tablet_service.cc:1467] Processing CreateTablet for tablet 44c9a15c2ed944aebff7390622aa8768 (DEFAULT_TABLE table=test-table [id=4a30dcb06cea4fdfba925f7dd695e630]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 22:47:58.520174 27364 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 44c9a15c2ed944aebff7390622aa8768. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:47:58.520810 27232 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 44c9a15c2ed944aebff7390622aa8768. 1 dirs total, 0 dirs full, 0 dirs failed
W20260925 22:47:58.523505 27160 tablet.cc:2369] T 44c9a15c2ed944aebff7390622aa8768 P 84c93dfd42c94d188c85a636289a4e23: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 22:47:58.527127 27190 tablet_bootstrap.cc:492] T 44c9a15c2ed944aebff7390622aa8768 P 1b6eadd191934a76ba315f4c414dd647: Bootstrap starting.
I20260925 22:47:58.529837 27190 tablet_bootstrap.cc:654] T 44c9a15c2ed944aebff7390622aa8768 P 1b6eadd191934a76ba315f4c414dd647: Neither blocks nor log segments found. Creating new log.
I20260925 22:47:58.530970 27190 log.cc:824] T 44c9a15c2ed944aebff7390622aa8768 P 1b6eadd191934a76ba315f4c414dd647: Log is configured to *not* fsync() on all Append() calls
I20260925 22:47:58.531289 27443 tablet_bootstrap.cc:492] T 44c9a15c2ed944aebff7390622aa8768 P e299c7a0d9b04d1a8d30de149122fc48: Bootstrap starting.
I20260925 22:47:58.533674 27190 tablet_bootstrap.cc:492] T 44c9a15c2ed944aebff7390622aa8768 P 1b6eadd191934a76ba315f4c414dd647: No bootstrap required, opened a new log
I20260925 22:47:58.533849 27443 tablet_bootstrap.cc:654] T 44c9a15c2ed944aebff7390622aa8768 P e299c7a0d9b04d1a8d30de149122fc48: Neither blocks nor log segments found. Creating new log.
I20260925 22:47:58.533996 27190 ts_tablet_manager.cc:1397] T 44c9a15c2ed944aebff7390622aa8768 P 1b6eadd191934a76ba315f4c414dd647: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260925 22:47:58.535162 27443 log.cc:824] T 44c9a15c2ed944aebff7390622aa8768 P e299c7a0d9b04d1a8d30de149122fc48: Log is configured to *not* fsync() on all Append() calls
I20260925 22:47:58.537853 27443 tablet_bootstrap.cc:492] T 44c9a15c2ed944aebff7390622aa8768 P e299c7a0d9b04d1a8d30de149122fc48: No bootstrap required, opened a new log
I20260925 22:47:58.538107 27443 ts_tablet_manager.cc:1397] T 44c9a15c2ed944aebff7390622aa8768 P e299c7a0d9b04d1a8d30de149122fc48: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20260925 22:47:58.537926 27190 raft_consensus.cc:348] T 44c9a15c2ed944aebff7390622aa8768 P 1b6eadd191934a76ba315f4c414dd647 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e299c7a0d9b04d1a8d30de149122fc48" member_type: VOTER last_known_addr { host: "127.10.198.3" port: 43405 } } peers { permanent_uuid: "1b6eadd191934a76ba315f4c414dd647" member_type: VOTER last_known_addr { host: "127.10.198.2" port: 40709 } } peers { permanent_uuid: "84c93dfd42c94d188c85a636289a4e23" member_type: VOTER last_known_addr { host: "127.10.198.1" port: 36769 } }
I20260925 22:47:58.538226 27190 raft_consensus.cc:374] T 44c9a15c2ed944aebff7390622aa8768 P 1b6eadd191934a76ba315f4c414dd647 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:47:58.538298 27190 raft_consensus.cc:729] T 44c9a15c2ed944aebff7390622aa8768 P 1b6eadd191934a76ba315f4c414dd647 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b6eadd191934a76ba315f4c414dd647, State: Initialized, Role: FOLLOWER
I20260925 22:47:58.538838 27190 consensus_queue.cc:260] T 44c9a15c2ed944aebff7390622aa8768 P 1b6eadd191934a76ba315f4c414dd647 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e299c7a0d9b04d1a8d30de149122fc48" member_type: VOTER last_known_addr { host: "127.10.198.3" port: 43405 } } peers { permanent_uuid: "1b6eadd191934a76ba315f4c414dd647" member_type: VOTER last_known_addr { host: "127.10.198.2" port: 40709 } } peers { permanent_uuid: "84c93dfd42c94d188c85a636289a4e23" member_type: VOTER last_known_addr { host: "127.10.198.1" port: 36769 } }
I20260925 22:47:58.539671 27298 heartbeater.cc:503] Master 127.10.198.62:41557 was elected leader, sending a full tablet report...
I20260925 22:47:58.540175 27190 ts_tablet_manager.cc:1428] T 44c9a15c2ed944aebff7390622aa8768 P 1b6eadd191934a76ba315f4c414dd647: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260925 22:47:58.542168 27443 raft_consensus.cc:348] T 44c9a15c2ed944aebff7390622aa8768 P e299c7a0d9b04d1a8d30de149122fc48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e299c7a0d9b04d1a8d30de149122fc48" member_type: VOTER last_known_addr { host: "127.10.198.3" port: 43405 } } peers { permanent_uuid: "1b6eadd191934a76ba315f4c414dd647" member_type: VOTER last_known_addr { host: "127.10.198.2" port: 40709 } } peers { permanent_uuid: "84c93dfd42c94d188c85a636289a4e23" member_type: VOTER last_known_addr { host: "127.10.198.1" port: 36769 } }
I20260925 22:47:58.542502 27443 raft_consensus.cc:374] T 44c9a15c2ed944aebff7390622aa8768 P e299c7a0d9b04d1a8d30de149122fc48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:47:58.542585 27443 raft_consensus.cc:729] T 44c9a15c2ed944aebff7390622aa8768 P e299c7a0d9b04d1a8d30de149122fc48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e299c7a0d9b04d1a8d30de149122fc48, State: Initialized, Role: FOLLOWER
I20260925 22:47:58.543116 27443 consensus_queue.cc:260] T 44c9a15c2ed944aebff7390622aa8768 P e299c7a0d9b04d1a8d30de149122fc48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e299c7a0d9b04d1a8d30de149122fc48" member_type: VOTER last_known_addr { host: "127.10.198.3" port: 43405 } } peers { permanent_uuid: "1b6eadd191934a76ba315f4c414dd647" member_type: VOTER last_known_addr { host: "127.10.198.2" port: 40709 } } peers { permanent_uuid: "84c93dfd42c94d188c85a636289a4e23" member_type: VOTER last_known_addr { host: "127.10.198.1" port: 36769 } }
I20260925 22:47:58.544101 27430 heartbeater.cc:503] Master 127.10.198.62:41557 was elected leader, sending a full tablet report...
I20260925 22:47:58.544440 27443 ts_tablet_manager.cc:1428] T 44c9a15c2ed944aebff7390622aa8768 P e299c7a0d9b04d1a8d30de149122fc48: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260925 22:47:58.574259 27165 raft_consensus.cc:482] T 44c9a15c2ed944aebff7390622aa8768 P 84c93dfd42c94d188c85a636289a4e23 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 22:47:58.574543 27165 raft_consensus.cc:504] T 44c9a15c2ed944aebff7390622aa8768 P 84c93dfd42c94d188c85a636289a4e23 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e299c7a0d9b04d1a8d30de149122fc48" member_type: VOTER last_known_addr { host: "127.10.198.3" port: 43405 } } peers { permanent_uuid: "1b6eadd191934a76ba315f4c414dd647" member_type: VOTER last_known_addr { host: "127.10.198.2" port: 40709 } } peers { permanent_uuid: "84c93dfd42c94d188c85a636289a4e23" member_type: VOTER last_known_addr { host: "127.10.198.1" port: 36769 } }
I20260925 22:47:58.576195 27165 leader_election.cc:288] T 44c9a15c2ed944aebff7390622aa8768 P 84c93dfd42c94d188c85a636289a4e23 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e299c7a0d9b04d1a8d30de149122fc48 (127.10.198.3:43405), 1b6eadd191934a76ba315f4c414dd647 (127.10.198.2:40709)
I20260925 22:47:58.581817 27441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:47:58.577074 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.198.1:42637 (local address 127.10.198.2:40709)
0925 22:47:58.577206 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0925 22:47:58.577212 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 22:47:58.577699 (+ 487us) server_negotiation.cc:407] Connection header received
0925 22:47:58.578042 (+ 343us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:47:58.578045 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:47:58.578106 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:47:58.578185 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:47:58.578653 (+ 468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:47:58.579362 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:47:58.580693 (+ 1331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:47:58.581132 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:47:58.581295 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:47:58.581352 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0925 22:47:58.581584 (+ 232us) server_negotiation.cc:299] Negotiation successful
0925 22:47:58.581668 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260925 22:47:58.581831 27439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:47:58.576638 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.198.1:60169 (local address 127.10.198.3:43405)
0925 22:47:58.576812 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0925 22:47:58.576819 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 22:47:58.578069 (+ 1250us) server_negotiation.cc:407] Connection header received
0925 22:47:58.578135 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:47:58.578138 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:47:58.578190 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:47:58.578266 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:47:58.578779 (+ 513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:47:58.579471 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:47:58.580692 (+ 1221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:47:58.581131 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:47:58.581286 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:47:58.581346 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0925 22:47:58.581625 (+ 279us) server_negotiation.cc:299] Negotiation successful
0925 22:47:58.581707 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260925 22:47:58.581835 27449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:47:58.576449 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.198.3:43405 (local address 127.10.198.1:60169)
0925 22:47:58.577264 (+ 815us) negotiation.cc:107] Waiting for socket to connect
0925 22:47:58.577308 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0925 22:47:58.577858 (+ 550us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:47:58.578402 (+ 544us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:47:58.578407 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:47:58.578427 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:47:58.578630 (+ 203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:47:58.578638 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:47:58.579599 (+ 961us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:47:58.579602 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:47:58.580536 (+ 934us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:47:58.580547 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:47:58.581274 (+ 727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:47:58.581278 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:47:58.581398 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:47:58.581488 (+ 90us) client_negotiation.cc:769] Sending connection context
0925 22:47:58.581586 (+ 98us) client_negotiation.cc:240] Negotiation successful
0925 22:47:58.581676 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":687,"mutex_wait_us":126,"thread_start_us":123,"threads_started":1}
I20260925 22:47:58.581853 27448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:47:58.576880 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.198.2:40709 (local address 127.10.198.1:42637)
0925 22:47:58.577595 (+ 715us) negotiation.cc:107] Waiting for socket to connect
0925 22:47:58.577611 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 22:47:58.577921 (+ 310us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:47:58.578334 (+ 413us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:47:58.578338 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:47:58.578359 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:47:58.578509 (+ 150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:47:58.578517 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:47:58.579469 (+ 952us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:47:58.579473 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:47:58.580535 (+ 1062us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:47:58.580547 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:47:58.581307 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:47:58.581310 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:47:58.581454 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:47:58.581473 (+ 19us) client_negotiation.cc:769] Sending connection context
0925 22:47:58.581536 (+ 63us) client_negotiation.cc:240] Negotiation successful
0925 22:47:58.581648 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":642,"mutex_wait_us":71,"thread_start_us":81,"threads_started":1}
I20260925 22:47:58.583058 27252 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44c9a15c2ed944aebff7390622aa8768" candidate_uuid: "84c93dfd42c94d188c85a636289a4e23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b6eadd191934a76ba315f4c414dd647" is_pre_election: true
I20260925 22:47:58.583048 27384 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44c9a15c2ed944aebff7390622aa8768" candidate_uuid: "84c93dfd42c94d188c85a636289a4e23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e299c7a0d9b04d1a8d30de149122fc48" is_pre_election: true
I20260925 22:47:58.583420 27384 raft_consensus.cc:2454] T 44c9a15c2ed944aebff7390622aa8768 P e299c7a0d9b04d1a8d30de149122fc48 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 84c93dfd42c94d188c85a636289a4e23 in term 0.
I20260925 22:47:58.583420 27252 raft_consensus.cc:2454] T 44c9a15c2ed944aebff7390622aa8768 P 1b6eadd191934a76ba315f4c414dd647 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 84c93dfd42c94d188c85a636289a4e23 in term 0.
I20260925 22:47:58.584050 27045 leader_election.cc:302] T 44c9a15c2ed944aebff7390622aa8768 P 84c93dfd42c94d188c85a636289a4e23 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b6eadd191934a76ba315f4c414dd647, 84c93dfd42c94d188c85a636289a4e23; no voters:
I20260925 22:47:58.584429 27165 raft_consensus.cc:2781] T 44c9a15c2ed944aebff7390622aa8768 P 84c93dfd42c94d188c85a636289a4e23 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 22:47:58.584578 27165 raft_consensus.cc:482] T 44c9a15c2ed944aebff7390622aa8768 P 84c93dfd42c94d188c85a636289a4e23 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 22:47:58.584673 27165 raft_consensus.cc:3037] T 44c9a15c2ed944aebff7390622aa8768 P 84c93dfd42c94d188c85a636289a4e23 [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:47:58.585896 27165 raft_consensus.cc:504] T 44c9a15c2ed944aebff7390622aa8768 P 84c93dfd42c94d188c85a636289a4e23 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e299c7a0d9b04d1a8d30de149122fc48" member_type: VOTER last_known_addr { host: "127.10.198.3" port: 43405 } } peers { permanent_uuid: "1b6eadd191934a76ba315f4c414dd647" member_type: VOTER last_known_addr { host: "127.10.198.2" port: 40709 } } peers { permanent_uuid: "84c93dfd42c94d188c85a636289a4e23" member_type: VOTER last_known_addr { host: "127.10.198.1" port: 36769 } }
I20260925 22:47:58.586374 27165 leader_election.cc:288] T 44c9a15c2ed944aebff7390622aa8768 P 84c93dfd42c94d188c85a636289a4e23 [CANDIDATE]: Term 1 election: Requested vote from peers e299c7a0d9b04d1a8d30de149122fc48 (127.10.198.3:43405), 1b6eadd191934a76ba315f4c414dd647 (127.10.198.2:40709)
I20260925 22:47:58.586961 27384 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44c9a15c2ed944aebff7390622aa8768" candidate_uuid: "84c93dfd42c94d188c85a636289a4e23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e299c7a0d9b04d1a8d30de149122fc48"
I20260925 22:47:58.587106 27252 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44c9a15c2ed944aebff7390622aa8768" candidate_uuid: "84c93dfd42c94d188c85a636289a4e23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b6eadd191934a76ba315f4c414dd647"
I20260925 22:47:58.587201 27384 raft_consensus.cc:3037] T 44c9a15c2ed944aebff7390622aa8768 P e299c7a0d9b04d1a8d30de149122fc48 [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:47:58.587285 27252 raft_consensus.cc:3037] T 44c9a15c2ed944aebff7390622aa8768 P 1b6eadd191934a76ba315f4c414dd647 [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:47:58.588613 27252 raft_consensus.cc:2454] T 44c9a15c2ed944aebff7390622aa8768 P 1b6eadd191934a76ba315f4c414dd647 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 84c93dfd42c94d188c85a636289a4e23 in term 1.
I20260925 22:47:58.588622 27384 raft_consensus.cc:2454] T 44c9a15c2ed944aebff7390622aa8768 P e299c7a0d9b04d1a8d30de149122fc48 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 84c93dfd42c94d188c85a636289a4e23 in term 1.
I20260925 22:47:58.589094 27045 leader_election.cc:302] T 44c9a15c2ed944aebff7390622aa8768 P 84c93dfd42c94d188c85a636289a4e23 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 84c93dfd42c94d188c85a636289a4e23, e299c7a0d9b04d1a8d30de149122fc48; no voters:
I20260925 22:47:58.589435 27165 raft_consensus.cc:2781] T 44c9a15c2ed944aebff7390622aa8768 P 84c93dfd42c94d188c85a636289a4e23 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 22:47:58.589684 27165 raft_consensus.cc:686] T 44c9a15c2ed944aebff7390622aa8768 P 84c93dfd42c94d188c85a636289a4e23 [term 1 LEADER]: Becoming Leader. State: Replica: 84c93dfd42c94d188c85a636289a4e23, State: Running, Role: LEADER
I20260925 22:47:58.589982 27165 consensus_queue.cc:237] T 44c9a15c2ed944aebff7390622aa8768 P 84c93dfd42c94d188c85a636289a4e23 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e299c7a0d9b04d1a8d30de149122fc48" member_type: VOTER last_known_addr { host: "127.10.198.3" port: 43405 } } peers { permanent_uuid: "1b6eadd191934a76ba315f4c414dd647" member_type: VOTER last_known_addr { host: "127.10.198.2" port: 40709 } } peers { permanent_uuid: "84c93dfd42c94d188c85a636289a4e23" member_type: VOTER last_known_addr { host: "127.10.198.1" port: 36769 } }
I20260925 22:47:58.593467 26970 catalog_manager.cc:5495] T 44c9a15c2ed944aebff7390622aa8768 P 84c93dfd42c94d188c85a636289a4e23 reported cstate change: term changed from 0 to 1, leader changed from <none> to 84c93dfd42c94d188c85a636289a4e23 (127.10.198.1). New cstate: current_term: 1 leader_uuid: "84c93dfd42c94d188c85a636289a4e23" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e299c7a0d9b04d1a8d30de149122fc48" member_type: VOTER last_known_addr { host: "127.10.198.3" port: 43405 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1b6eadd191934a76ba315f4c414dd647" member_type: VOTER last_known_addr { host: "127.10.198.2" port: 40709 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "84c93dfd42c94d188c85a636289a4e23" member_type: VOTER last_known_addr { host: "127.10.198.1" port: 36769 } health_report { overall_health: HEALTHY } } }
I20260925 22:47:58.625515 27453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:47:58.621017 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59430 (local address 127.10.198.1:36769)
0925 22:47:58.621353 (+ 336us) server_negotiation.cc:206] Beginning negotiation
0925 22:47:58.621361 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 22:47:58.621376 (+ 15us) server_negotiation.cc:407] Connection header received
0925 22:47:58.621463 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:47:58.621465 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:47:58.621528 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:47:58.621654 (+ 126us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 22:47:58.621887 (+ 233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:47:58.622554 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:47:58.623603 (+ 1049us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:47:58.623975 (+ 372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:47:58.624275 (+ 300us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:47:58.624488 (+ 213us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 22:47:58.624990 (+ 502us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 22:47:58.625102 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0925 22:47:58.625200 (+ 98us) server_negotiation.cc:299] Negotiation successful
0925 22:47:58.625265 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":257,"thread_start_us":154,"threads_started":1}
I20260925 22:47:58.630692 27252 raft_consensus.cc:1261] T 44c9a15c2ed944aebff7390622aa8768 P 1b6eadd191934a76ba315f4c414dd647 [term 1 FOLLOWER]: Refusing update from remote peer 84c93dfd42c94d188c85a636289a4e23: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 22:47:58.630877 27384 raft_consensus.cc:1261] T 44c9a15c2ed944aebff7390622aa8768 P e299c7a0d9b04d1a8d30de149122fc48 [term 1 FOLLOWER]: Refusing update from remote peer 84c93dfd42c94d188c85a636289a4e23: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 22:47:58.631913 27450 consensus_queue.cc:1035] T 44c9a15c2ed944aebff7390622aa8768 P 84c93dfd42c94d188c85a636289a4e23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b6eadd191934a76ba315f4c414dd647" member_type: VOTER last_known_addr { host: "127.10.198.2" port: 40709 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 22:47:58.632261 27165 consensus_queue.cc:1035] T 44c9a15c2ed944aebff7390622aa8768 P 84c93dfd42c94d188c85a636289a4e23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e299c7a0d9b04d1a8d30de149122fc48" member_type: VOTER last_known_addr { host: "127.10.198.3" port: 43405 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 22:47:58.642474 27455 mvcc.cc:205] Tried to move back new op lower bound from 7333382056462147584 to 7333382056307781632. Current Snapshot: MvccSnapshot[applied={T|T < 7333382056462147584}]
I20260925 22:47:58.653708 26970 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39378:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 22:47:58.653900 26970 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39378:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 22:47:58.656786 26970 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3f1861b6350a4355939000f5afa487a5: Sending DeleteTablet for 3 replicas of tablet 44c9a15c2ed944aebff7390622aa8768
I20260925 22:47:58.657763 27093 tablet_service.cc:1514] Processing DeleteTablet for tablet 44c9a15c2ed944aebff7390622aa8768 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 22:47:58 UTC) from {username='kudu', principal='kudu/127.10.198.62@KRBTEST.COM'} at 127.0.0.1:59414
I20260925 22:47:58.657707 27364 tablet_service.cc:1514] Processing DeleteTablet for tablet 44c9a15c2ed944aebff7390622aa8768 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 22:47:58 UTC) from {username='kudu', principal='kudu/127.10.198.62@KRBTEST.COM'} at 127.0.0.1:33992
I20260925 22:47:58.657706 27232 tablet_service.cc:1514] Processing DeleteTablet for tablet 44c9a15c2ed944aebff7390622aa8768 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 22:47:58 UTC) from {username='kudu', principal='kudu/127.10.198.62@KRBTEST.COM'} at 127.0.0.1:54768
I20260925 22:47:58.658056 11032 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrWXm8X/build/debug/bin/kudu with pid 27009
I20260925 22:47:58.658478 27466 tablet_replica.cc:330] T 44c9a15c2ed944aebff7390622aa8768 P 1b6eadd191934a76ba315f4c414dd647: stopping tablet replica
I20260925 22:47:58.658335 27465 tablet_replica.cc:330] T 44c9a15c2ed944aebff7390622aa8768 P e299c7a0d9b04d1a8d30de149122fc48: stopping tablet replica
I20260925 22:47:58.659057 27465 raft_consensus.cc:2229] T 44c9a15c2ed944aebff7390622aa8768 P e299c7a0d9b04d1a8d30de149122fc48 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 22:47:58.659058 27466 raft_consensus.cc:2229] T 44c9a15c2ed944aebff7390622aa8768 P 1b6eadd191934a76ba315f4c414dd647 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 22:47:58.659461 27465 raft_consensus.cc:2258] T 44c9a15c2ed944aebff7390622aa8768 P e299c7a0d9b04d1a8d30de149122fc48 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 22:47:58.659493 27466 raft_consensus.cc:2258] T 44c9a15c2ed944aebff7390622aa8768 P 1b6eadd191934a76ba315f4c414dd647 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 22:47:58.661311 27466 ts_tablet_manager.cc:1905] T 44c9a15c2ed944aebff7390622aa8768 P 1b6eadd191934a76ba315f4c414dd647: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 22:47:58.661368 27465 ts_tablet_manager.cc:1905] T 44c9a15c2ed944aebff7390622aa8768 P e299c7a0d9b04d1a8d30de149122fc48: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 22:47:58.663954 27466 ts_tablet_manager.cc:1918] T 44c9a15c2ed944aebff7390622aa8768 P 1b6eadd191934a76ba315f4c414dd647: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 22:47:58.664079 27466 log.cc:1196] T 44c9a15c2ed944aebff7390622aa8768 P 1b6eadd191934a76ba315f4c414dd647: Deleting WAL directory at /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376259650642-11032-0/minicluster-data/ts-1/wal/wals/44c9a15c2ed944aebff7390622aa8768
I20260925 22:47:58.664382 27466 ts_tablet_manager.cc:1939] T 44c9a15c2ed944aebff7390622aa8768 P 1b6eadd191934a76ba315f4c414dd647: Deleting consensus metadata
I20260925 22:47:58.665269 27465 ts_tablet_manager.cc:1918] T 44c9a15c2ed944aebff7390622aa8768 P e299c7a0d9b04d1a8d30de149122fc48: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 22:47:58.665405 27465 log.cc:1196] T 44c9a15c2ed944aebff7390622aa8768 P e299c7a0d9b04d1a8d30de149122fc48: Deleting WAL directory at /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376259650642-11032-0/minicluster-data/ts-2/wal/wals/44c9a15c2ed944aebff7390622aa8768
I20260925 22:47:58.665817 27465 ts_tablet_manager.cc:1939] T 44c9a15c2ed944aebff7390622aa8768 P e299c7a0d9b04d1a8d30de149122fc48: Deleting consensus metadata
I20260925 22:47:58.665856 26955 catalog_manager.cc:4841] TS 1b6eadd191934a76ba315f4c414dd647 (127.10.198.2:40709): tablet 44c9a15c2ed944aebff7390622aa8768 (table test-table [id=4a30dcb06cea4fdfba925f7dd695e630]) successfully deleted
W20260925 22:47:58.666982 26955 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.10.198.1:36769 (error 108)
I20260925 22:47:58.667506 26955 catalog_manager.cc:4841] TS e299c7a0d9b04d1a8d30de149122fc48 (127.10.198.3:43405): tablet 44c9a15c2ed944aebff7390622aa8768 (table test-table [id=4a30dcb06cea4fdfba925f7dd695e630]) successfully deleted
I20260925 22:47:58.668187 27440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:47:58.667833 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.198.1:36769 (local address 127.0.0.1:59444)
0925 22:47:58.668052 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0925 22:47:58.668111 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.198.1:36769: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":152}
I20260925 22:47:58.668180 11032 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrWXm8X/build/debug/bin/kudu with pid 27164
W20260925 22:47:58.668409 26955 catalog_manager.cc:4584] TS 84c93dfd42c94d188c85a636289a4e23 (127.10.198.1:36769): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 44c9a15c2ed944aebff7390622aa8768: Network error: Client connection negotiation failed: client connection to 127.10.198.1:36769: connect: Connection refused (error 111)
I20260925 22:47:58.675015 11032 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrWXm8X/build/debug/bin/kudu with pid 27301
I20260925 22:47:58.681100 11032 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrWXm8X/build/debug/bin/kudu with pid 26938
I20260925 22:48:18.688238 11032 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrWXm8X/build/debug/bin/kudu
/tmp/dist-test-taskrWXm8X/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376259650642-11032-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376259650642-11032-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376259650642-11032-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376259650642-11032-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.198.62:41557
--webserver_interface=127.10.198.62
--webserver_port=35045
--keytab_file=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376259650642-11032-0/krb5kdc/kudu_127.10.198.62.keytab
--principal=kudu/127.10.198.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.198.20:41079
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.198.62:41557
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376259650642-11032-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376259650642-11032-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376259650642-11032-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:48:18.806915 27468 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:48:18.807303 27468 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:48:18.807395 27468 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:48:18.811784 27468 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260925 22:48:18.811940 27468 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260925 22:48:18.811967 27468 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260925 22:48:18.811990 27468 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 22:48:18.812036 27468 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:48:18.812078 27468 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:48:18.812136 27468 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 22:48:18.812175 27468 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 22:48:18.818070 27468 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.198.20:41079
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376259650642-11032-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376259650642-11032-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.10.198.62:41557
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376259650642-11032-0/krb5kdc/kudu_127.10.198.62.keytab
--principal=kudu/127.10.198.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.198.62:41557
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376259650642-11032-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.198.62
--webserver_port=35045
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27468
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376259650642-11032-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:09:16 UTC on ccycloud.quasar-afmhyw.root.comops.site
build id 4673379
I20260925 22:48:18.819319 27468 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:48:18.820461 27468 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 22:48:18.830057 27468 server_base.cc:1034] running on GCE node
W20260925 22:48:18.830026 27473 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:48:18.830063 27474 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:48:18.830087 27476 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:48:18.831061 27468 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:48:18.832410 27468 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:48:18.833635 27468 hybrid_clock.cc:648] HybridClock initialized: now 1790376498833604 us; error 51 us; skew 500 ppm
Sep 25 22:48:18 dist-test-slave-5vdr krb5kdc[26391](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376498, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.198.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:48:18.837122 27468 init.cc:377] Logged in from keytab as kudu/127.10.198.62@KRBTEST.COM (short username kudu)
I20260925 22:48:18.838183 27468 webserver.cc:460] Webserver started at http://127.10.198.62:35045/ using document root <none> and password file <none>
I20260925 22:48:18.838851 27468 fs_manager.cc:359] Metadata directory not provided
I20260925 22:48:18.838964 27468 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:48:18.843482 27468 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260925 22:48:18.846140 27483 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:48:18.847699 27468 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260925 22:48:18.847945 27468 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376259650642-11032-0/minicluster-data/master-0/data,/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376259650642-11032-0/minicluster-data/master-0/wal
uuid: "3f1861b6350a4355939000f5afa487a5"
format_stamp: "Formatted at 2026-09-25 22:47:36 on dist-test-slave-5vdr"
server_key: "3e784eed9d2d48329bac4e54677457fc"
server_key_iv: "5da66deeb4410b5f5519f8ab958f454f"
server_key_version: "encryptionkey@0"
I20260925 22:48:18.848594 27468 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376259650642-11032-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376259650642-11032-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376259650642-11032-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:48:18.864722 27468 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:48:18.868526 27468 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:48:18.868764 27468 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:48:18.876528 27468 rpc_server.cc:307] RPC server started. Bound to: 127.10.198.62:41557
I20260925 22:48:18.876821 27535 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.198.62:41557 every 8 connection(s)
I20260925 22:48:18.877784 27468 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376259650642-11032-0/minicluster-data/master-0/data/info.pb
I20260925 22:48:18.882472 27536 sys_catalog.cc:263] Verifying existing consensus state
I20260925 22:48:18.884539 27536 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3f1861b6350a4355939000f5afa487a5: Bootstrap starting.
I20260925 22:48:18.886157 11032 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrWXm8X/build/debug/bin/kudu as pid 27468
I20260925 22:48:18.886652 11032 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrWXm8X/build/debug/bin/kudu
/tmp/dist-test-taskrWXm8X/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376259650642-11032-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376259650642-11032-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376259650642-11032-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376259650642-11032-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.198.1:36769
--local_ip_for_outbound_sockets=127.10.198.1
--tserver_master_addrs=127.10.198.62:41557
--webserver_port=34609
--webserver_interface=127.10.198.1
--keytab_file=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376259650642-11032-0/krb5kdc/kudu_127.10.198.1.keytab
--principal=kudu/127.10.198.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.198.20:41079
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376259650642-11032-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376259650642-11032-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376259650642-11032-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 22:48:18.909961 27536 log.cc:824] T 00000000000000000000000000000000 P 3f1861b6350a4355939000f5afa487a5: Log is configured to *not* fsync() on all Append() calls
I20260925 22:48:18.926221 27536 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3f1861b6350a4355939000f5afa487a5: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 22:48:18.926745 27536 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3f1861b6350a4355939000f5afa487a5: Bootstrap complete.
I20260925 22:48:18.932482 27536 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3f1861b6350a4355939000f5afa487a5 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f1861b6350a4355939000f5afa487a5" member_type: VOTER last_known_addr { host: "127.10.198.62" port: 41557 } }
I20260925 22:48:18.933066 27536 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3f1861b6350a4355939000f5afa487a5 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3f1861b6350a4355939000f5afa487a5, State: Initialized, Role: FOLLOWER
I20260925 22:48:18.933699 27536 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3f1861b6350a4355939000f5afa487a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f1861b6350a4355939000f5afa487a5" member_type: VOTER last_known_addr { host: "127.10.198.62" port: 41557 } }
I20260925 22:48:18.933871 27536 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3f1861b6350a4355939000f5afa487a5 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 22:48:18.933938 27536 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3f1861b6350a4355939000f5afa487a5 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 22:48:18.934048 27536 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3f1861b6350a4355939000f5afa487a5 [term 2 FOLLOWER]: Advancing to term 3
I20260925 22:48:18.936967 27536 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3f1861b6350a4355939000f5afa487a5 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f1861b6350a4355939000f5afa487a5" member_type: VOTER last_known_addr { host: "127.10.198.62" port: 41557 } }
I20260925 22:48:18.937398 27536 leader_election.cc:302] T 00000000000000000000000000000000 P 3f1861b6350a4355939000f5afa487a5 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3f1861b6350a4355939000f5afa487a5; no voters:
I20260925 22:48:18.937775 27536 leader_election.cc:288] T 00000000000000000000000000000000 P 3f1861b6350a4355939000f5afa487a5 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260925 22:48:18.938006 27540 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3f1861b6350a4355939000f5afa487a5 [term 3 FOLLOWER]: Leader election won for term 3
I20260925 22:48:18.938344 27540 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3f1861b6350a4355939000f5afa487a5 [term 3 LEADER]: Becoming Leader. State: Replica: 3f1861b6350a4355939000f5afa487a5, State: Running, Role: LEADER
I20260925 22:48:18.938733 27540 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3f1861b6350a4355939000f5afa487a5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f1861b6350a4355939000f5afa487a5" member_type: VOTER last_known_addr { host: "127.10.198.62" port: 41557 } }
I20260925 22:48:18.939257 27536 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3f1861b6350a4355939000f5afa487a5 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 22:48:18.940492 27542 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3f1861b6350a4355939000f5afa487a5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3f1861b6350a4355939000f5afa487a5. Latest consensus state: current_term: 3 leader_uuid: "3f1861b6350a4355939000f5afa487a5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f1861b6350a4355939000f5afa487a5" member_type: VOTER last_known_addr { host: "127.10.198.62" port: 41557 } } }
I20260925 22:48:18.940605 27542 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3f1861b6350a4355939000f5afa487a5 [sys.catalog]: This master's current role is: LEADER
I20260925 22:48:18.940445 27541 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3f1861b6350a4355939000f5afa487a5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "3f1861b6350a4355939000f5afa487a5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f1861b6350a4355939000f5afa487a5" member_type: VOTER last_known_addr { host: "127.10.198.62" port: 41557 } } }
I20260925 22:48:18.940973 27541 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3f1861b6350a4355939000f5afa487a5 [sys.catalog]: This master's current role is: LEADER
I20260925 22:48:18.945288 27555 catalog_manager.cc:1259] Loaded cluster ID: ff32c441e4cc4360a3fe479cb6485ef5
I20260925 22:48:18.945384 27555 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 3f1861b6350a4355939000f5afa487a5: loading cluster ID for follower catalog manager: success
I20260925 22:48:18.948153 27555 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 3f1861b6350a4355939000f5afa487a5: acquiring CA information for follower catalog manager: success
F20260925 22:48:18.949673 27555 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7ff5769379ad google::LogMessage::Fail() at ??:0
@ 0x7ff57693a88a google::LogMessage::SendToLog() at ??:0
@ 0x7ff5769374de google::LogMessage::Flush() at ??:0
@ 0x7ff576938a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7ff5795fb18d kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7ff58129c989 kudu::master::CatalogManager::LoadTspkEntries() at ??:0
@ 0x7ff581279023 kudu::master::CatalogManager::PrepareFollowerTokenVerifier() at ??:0
@ 0x7ff58127954b kudu::master::CatalogManager::PrepareFollower() at ??:0
@ 0x7ff581272652 kudu::master::CatalogManagerBgTasks::Run() at ??:0
@ 0x7ff5812719e9 _ZZN4kudu6master21CatalogManagerBgTasks4InitEvENKUlvE_clEv at ??:0
@ 0x7ff5812b8541 _ZNSt17_Function_handlerIFvvEZN4kudu6master21CatalogManagerBgTasks4InitEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7ff578b7a512 kudu::Thread::SuperviseThread() at ??:0
@ 0x7ff57a1be6db start_thread at ??:0
@ 0x7ff57543561f clone at ??:0
W20260925 22:48:19.061398 27538 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:48:19.061859 27538 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:48:19.061944 27538 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:48:19.068625 27538 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:48:19.068744 27538 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:48:19.068845 27538 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.198.1
I20260925 22:48:19.075066 27538 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.198.20:41079
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376259650642-11032-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376259650642-11032-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376259650642-11032-0/krb5kdc/kudu_127.10.198.1.keytab
--principal=kudu/127.10.198.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.198.1:36769
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376259650642-11032-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.198.1
--webserver_port=34609
--tserver_master_addrs=127.10.198.62:41557
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27538
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.198.1
--log_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376259650642-11032-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:09:16 UTC on ccycloud.quasar-afmhyw.root.comops.site
build id 4673379
I20260925 22:48:19.076658 27538 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:48:19.077816 27538 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:48:19.088306 27576 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:48:19.088327 27577 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:48:19.088562 27538 server_base.cc:1034] running on GCE node
W20260925 22:48:19.088306 27579 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:48:19.089157 27538 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:48:19.089952 27538 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:48:19.091156 27538 hybrid_clock.cc:648] HybridClock initialized: now 1790376499091121 us; error 52 us; skew 500 ppm
Sep 25 22:48:19 dist-test-slave-5vdr krb5kdc[26391](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376499, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.198.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:48:19.094456 27538 init.cc:377] Logged in from keytab as kudu/127.10.198.1@KRBTEST.COM (short username kudu)
I20260925 22:48:19.095549 27538 webserver.cc:460] Webserver started at http://127.10.198.1:34609/ using document root <none> and password file <none>
I20260925 22:48:19.096246 27538 fs_manager.cc:359] Metadata directory not provided
I20260925 22:48:19.096349 27538 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:48:19.100893 27538 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260925 22:48:19.103538 27586 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:48:19.105316 27538 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 22:48:19.105477 27538 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376259650642-11032-0/minicluster-data/ts-0/data,/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376259650642-11032-0/minicluster-data/ts-0/wal
uuid: "84c93dfd42c94d188c85a636289a4e23"
format_stamp: "Formatted at 2026-09-25 22:47:36 on dist-test-slave-5vdr"
server_key: "e0c3fa3a013ecdfc35c9ebe8f8f04413"
server_key_iv: "c47103792e3bc31abe6ea5c9c990e8e2"
server_key_version: "encryptionkey@0"
I20260925 22:48:19.106101 27538 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376259650642-11032-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376259650642-11032-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376259650642-11032-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:48:19.128962 27538 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:48:19.132934 27538 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:48:19.133216 27538 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:48:19.134121 27538 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 22:48:19.135958 27593 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260925 22:48:19.142071 27538 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260925 22:48:19.142207 27538 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.007s user 0.001s sys 0.000s
I20260925 22:48:19.142357 27538 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260925 22:48:19.144460 27538 ts_tablet_manager.cc:610] Registered 1 tablets
I20260925 22:48:19.144556 27538 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.002s sys 0.000s
I20260925 22:48:19.144907 27593 tablet_bootstrap.cc:492] T 44c9a15c2ed944aebff7390622aa8768 P 84c93dfd42c94d188c85a636289a4e23: Bootstrap starting.
I20260925 22:48:19.156399 27538 rpc_server.cc:307] RPC server started. Bound to: 127.10.198.1:36769
I20260925 22:48:19.156744 27700 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.198.1:36769 every 8 connection(s)
I20260925 22:48:19.157603 27538 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376259650642-11032-0/minicluster-data/ts-0/data/info.pb
I20260925 22:48:19.166348 11032 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrWXm8X/build/debug/bin/kudu as pid 27538
I20260925 22:48:19.166859 11032 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrWXm8X/build/debug/bin/kudu
/tmp/dist-test-taskrWXm8X/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376259650642-11032-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376259650642-11032-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376259650642-11032-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376259650642-11032-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.198.2:40709
--local_ip_for_outbound_sockets=127.10.198.2
--tserver_master_addrs=127.10.198.62:41557
--webserver_port=40329
--webserver_interface=127.10.198.2
--keytab_file=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376259650642-11032-0/krb5kdc/kudu_127.10.198.2.keytab
--principal=kudu/127.10.198.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.198.20:41079
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376259650642-11032-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376259650642-11032-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376259650642-11032-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 22:48:19.169040 27593 log.cc:824] T 44c9a15c2ed944aebff7390622aa8768 P 84c93dfd42c94d188c85a636289a4e23: Log is configured to *not* fsync() on all Append() calls
I20260925 22:48:19.179888 27593 tablet_bootstrap.cc:492] T 44c9a15c2ed944aebff7390622aa8768 P 84c93dfd42c94d188c85a636289a4e23: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 22:48:19.180384 27593 tablet_bootstrap.cc:492] T 44c9a15c2ed944aebff7390622aa8768 P 84c93dfd42c94d188c85a636289a4e23: Bootstrap complete.
I20260925 22:48:19.181833 27593 ts_tablet_manager.cc:1397] T 44c9a15c2ed944aebff7390622aa8768 P 84c93dfd42c94d188c85a636289a4e23: Time spent bootstrapping tablet: real 0.037s user 0.027s sys 0.007s
I20260925 22:48:19.185459 27593 raft_consensus.cc:348] T 44c9a15c2ed944aebff7390622aa8768 P 84c93dfd42c94d188c85a636289a4e23 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e299c7a0d9b04d1a8d30de149122fc48" member_type: VOTER last_known_addr { host: "127.10.198.3" port: 43405 } } peers { permanent_uuid: "1b6eadd191934a76ba315f4c414dd647" member_type: VOTER last_known_addr { host: "127.10.198.2" port: 40709 } } peers { permanent_uuid: "84c93dfd42c94d188c85a636289a4e23" member_type: VOTER last_known_addr { host: "127.10.198.1" port: 36769 } }
I20260925 22:48:19.186088 27593 raft_consensus.cc:729] T 44c9a15c2ed944aebff7390622aa8768 P 84c93dfd42c94d188c85a636289a4e23 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 84c93dfd42c94d188c85a636289a4e23, State: Initialized, Role: FOLLOWER
I20260925 22:48:19.186681 27593 consensus_queue.cc:260] T 44c9a15c2ed944aebff7390622aa8768 P 84c93dfd42c94d188c85a636289a4e23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e299c7a0d9b04d1a8d30de149122fc48" member_type: VOTER last_known_addr { host: "127.10.198.3" port: 43405 } } peers { permanent_uuid: "1b6eadd191934a76ba315f4c414dd647" member_type: VOTER last_known_addr { host: "127.10.198.2" port: 40709 } } peers { permanent_uuid: "84c93dfd42c94d188c85a636289a4e23" member_type: VOTER last_known_addr { host: "127.10.198.1" port: 36769 } }
I20260925 22:48:19.187983 27593 ts_tablet_manager.cc:1428] T 44c9a15c2ed944aebff7390622aa8768 P 84c93dfd42c94d188c85a636289a4e23: Time spent starting tablet: real 0.006s user 0.007s sys 0.001s
W20260925 22:48:19.210971 27703 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:48:19.159768 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.198.62:41557 (local address 127.10.198.1:44327)
0925 22:48:19.160419 (+ 651us) negotiation.cc:107] Waiting for socket to connect
0925 22:48:19.160484 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0925 22:48:19.161361 (+ 877us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:48:19.209882 (+ 48521us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.198.62:41557: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":422,"thread_start_us":178,"threads_started":1}
W20260925 22:48:19.211561 27701 heartbeater.cc:644] Failed to heartbeat to 127.10.198.62:41557 (0 consecutive failures): Network error: Failed to ping master at 127.10.198.62:41557: Client connection negotiation failed: client connection to 127.10.198.62:41557: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260925 22:48:19.212548 27703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:48:19.212219 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.198.62:41557 (local address 127.10.198.1:57367)
0925 22:48:19.212376 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0925 22:48:19.212466 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.198.62:41557: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260925 22:48:19.213418 27703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:48:19.213164 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.198.62:41557 (local address 127.10.198.1:36479)
0925 22:48:19.213290 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0925 22:48:19.213345 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.198.62:41557: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260925 22:48:19.213681 27701 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260925 22:48:19.293010 27704 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:48:19.293329 27704 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:48:19.293376 27704 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:48:19.297789 27704 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:48:19.297884 27704 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:48:19.297950 27704 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.198.2
I20260925 22:48:19.303715 27704 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.198.20:41079
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376259650642-11032-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376259650642-11032-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376259650642-11032-0/krb5kdc/kudu_127.10.198.2.keytab
--principal=kudu/127.10.198.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.198.2:40709
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376259650642-11032-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.198.2
--webserver_port=40329
--tserver_master_addrs=127.10.198.62:41557
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27704
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.198.2
--log_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376259650642-11032-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:09:16 UTC on ccycloud.quasar-afmhyw.root.comops.site
build id 4673379
I20260925 22:48:19.304996 27704 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:48:19.306152 27704 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:48:19.316553 27711 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:48:19.316737 27714 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:48:19.316560 27712 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:48:19.316637 27704 server_base.cc:1034] running on GCE node
I20260925 22:48:19.317302 27704 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:48:19.318094 27704 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:48:19.319273 27704 hybrid_clock.cc:648] HybridClock initialized: now 1790376499319264 us; error 43 us; skew 500 ppm
Sep 25 22:48:19 dist-test-slave-5vdr krb5kdc[26391](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376499, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.198.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:48:19.322376 27704 init.cc:377] Logged in from keytab as kudu/127.10.198.2@KRBTEST.COM (short username kudu)
I20260925 22:48:19.323382 27704 webserver.cc:460] Webserver started at http://127.10.198.2:40329/ using document root <none> and password file <none>
I20260925 22:48:19.324095 27704 fs_manager.cc:359] Metadata directory not provided
I20260925 22:48:19.324175 27704 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:48:19.328503 27704 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.001s sys 0.002s
I20260925 22:48:19.330838 27721 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:48:19.332336 27704 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 22:48:19.332520 27704 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376259650642-11032-0/minicluster-data/ts-1/data,/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376259650642-11032-0/minicluster-data/ts-1/wal
uuid: "1b6eadd191934a76ba315f4c414dd647"
format_stamp: "Formatted at 2026-09-25 22:47:36 on dist-test-slave-5vdr"
server_key: "5f931a5901e7de1154c19256f2a62735"
server_key_iv: "38c9689b483c352210f94e845e378f22"
server_key_version: "encryptionkey@0"
I20260925 22:48:19.333353 27704 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376259650642-11032-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376259650642-11032-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376259650642-11032-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:48:19.363091 27704 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:48:19.367069 27704 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:48:19.367339 27704 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:48:19.368031 27704 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 22:48:19.368934 27704 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:48:19.369012 27704 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:48:19.369086 27704 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:48:19.369130 27704 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 22:48:19.378755 27704 rpc_server.cc:307] RPC server started. Bound to: 127.10.198.2:40709
I20260925 22:48:19.378788 27834 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.198.2:40709 every 8 connection(s)
I20260925 22:48:19.380046 27704 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376259650642-11032-0/minicluster-data/ts-1/data/info.pb
I20260925 22:48:19.384364 27837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:48:19.382442 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.198.62:41557 (local address 127.10.198.2:53559)
0925 22:48:19.382951 (+ 509us) negotiation.cc:107] Waiting for socket to connect
0925 22:48:19.383515 (+ 564us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.198.62:41557: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":292,"thread_start_us":143,"threads_started":1}
W20260925 22:48:19.384867 27835 heartbeater.cc:644] Failed to heartbeat to 127.10.198.62:41557 (0 consecutive failures): Network error: Failed to ping master at 127.10.198.62:41557: Client connection negotiation failed: client connection to 127.10.198.62:41557: connect: Connection refused (error 111)
I20260925 22:48:19.385582 27837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:48:19.385283 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.198.62:41557 (local address 127.10.198.2:60303)
0925 22:48:19.385421 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0925 22:48:19.385481 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.198.62:41557: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260925 22:48:19.385653 11032 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrWXm8X/build/debug/bin/kudu as pid 27704
I20260925 22:48:19.386171 11032 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrWXm8X/build/debug/bin/kudu
/tmp/dist-test-taskrWXm8X/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376259650642-11032-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376259650642-11032-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376259650642-11032-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376259650642-11032-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.198.3:43405
--local_ip_for_outbound_sockets=127.10.198.3
--tserver_master_addrs=127.10.198.62:41557
--webserver_port=43119
--webserver_interface=127.10.198.3
--keytab_file=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376259650642-11032-0/krb5kdc/kudu_127.10.198.3.keytab
--principal=kudu/127.10.198.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.198.20:41079
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376259650642-11032-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376259650642-11032-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376259650642-11032-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 22:48:19.386340 27837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:48:19.386077 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.198.62:41557 (local address 127.10.198.2:35967)
0925 22:48:19.386187 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0925 22:48:19.386230 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.198.62:41557: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260925 22:48:19.386475 27835 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260925 22:48:19.504859 27838 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:48:19.505167 27838 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:48:19.505247 27838 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:48:19.509591 27838 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:48:19.509670 27838 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:48:19.509769 27838 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.198.3
I20260925 22:48:19.518530 27838 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.198.20:41079
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376259650642-11032-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376259650642-11032-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376259650642-11032-0/krb5kdc/kudu_127.10.198.3.keytab
--principal=kudu/127.10.198.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.198.3:43405
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376259650642-11032-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.198.3
--webserver_port=43119
--tserver_master_addrs=127.10.198.62:41557
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27838
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.198.3
--log_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376259650642-11032-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:09:16 UTC on ccycloud.quasar-afmhyw.root.comops.site
build id 4673379
I20260925 22:48:19.520324 27838 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:48:19.521775 27838 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:48:19.532948 27844 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:48:19.533074 27838 server_base.cc:1034] running on GCE node
W20260925 22:48:19.532933 27846 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:48:19.532935 27843 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:48:19.533692 27838 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:48:19.534601 27838 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:48:19.535812 27838 hybrid_clock.cc:648] HybridClock initialized: now 1790376499535800 us; error 37 us; skew 500 ppm
Sep 25 22:48:19 dist-test-slave-5vdr krb5kdc[26391](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376499, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.198.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:48:19.539140 27838 init.cc:377] Logged in from keytab as kudu/127.10.198.3@KRBTEST.COM (short username kudu)
I20260925 22:48:19.540253 27838 webserver.cc:460] Webserver started at http://127.10.198.3:43119/ using document root <none> and password file <none>
I20260925 22:48:19.540938 27838 fs_manager.cc:359] Metadata directory not provided
I20260925 22:48:19.541065 27838 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:48:19.545497 27838 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260925 22:48:19.548048 27853 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:48:19.549444 27838 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 22:48:19.549630 27838 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376259650642-11032-0/minicluster-data/ts-2/data,/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376259650642-11032-0/minicluster-data/ts-2/wal
uuid: "e299c7a0d9b04d1a8d30de149122fc48"
format_stamp: "Formatted at 2026-09-25 22:47:37 on dist-test-slave-5vdr"
server_key: "5e2b92cea1a43418998f5092402082b4"
server_key_iv: "8e5ef74a1b33c58e90f390b421cc177c"
server_key_version: "encryptionkey@0"
I20260925 22:48:19.550262 27838 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376259650642-11032-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376259650642-11032-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376259650642-11032-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:48:19.590194 27838 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:48:19.593673 27838 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:48:19.593911 27838 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:48:19.594587 27838 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 22:48:19.595520 27838 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:48:19.595599 27838 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:48:19.595679 27838 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:48:19.595722 27838 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 22:48:19.605981 27838 rpc_server.cc:307] RPC server started. Bound to: 127.10.198.3:43405
I20260925 22:48:19.606024 27966 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.198.3:43405 every 8 connection(s)
I20260925 22:48:19.607134 27838 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790376259650642-11032-0/minicluster-data/ts-2/data/info.pb
I20260925 22:48:19.611384 27969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:48:19.609518 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.198.62:41557 (local address 127.10.198.3:41959)
0925 22:48:19.610033 (+ 515us) negotiation.cc:107] Waiting for socket to connect
0925 22:48:19.610586 (+ 553us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.198.62:41557: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":297,"thread_start_us":118,"threads_started":1}
W20260925 22:48:19.611968 27967 heartbeater.cc:644] Failed to heartbeat to 127.10.198.62:41557 (0 consecutive failures): Network error: Failed to ping master at 127.10.198.62:41557: Client connection negotiation failed: client connection to 127.10.198.62:41557: connect: Connection refused (error 111)
I20260925 22:48:19.612686 27969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:48:19.612432 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.198.62:41557 (local address 127.10.198.3:45991)
0925 22:48:19.612567 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0925 22:48:19.612624 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.198.62:41557: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260925 22:48:19.613412 27969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:48:19.613180 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.198.62:41557 (local address 127.10.198.3:58363)
0925 22:48:19.613300 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0925 22:48:19.613356 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.198.62:41557: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
W20260925 22:48:19.613611 27967 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260925 22:48:19.614306 11032 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrWXm8X/build/debug/bin/kudu as pid 27838
I20260925 22:48:20.215001 27971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:48:20.214474 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.198.62:41557 (local address 127.10.198.1:47341)
0925 22:48:20.214828 (+ 354us) negotiation.cc:107] Waiting for socket to connect
0925 22:48:20.214897 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.198.62:41557: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":226,"thread_start_us":81,"threads_started":1}
I20260925 22:48:20.387645 27972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:48:20.387182 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.198.62:41557 (local address 127.10.198.2:58049)
0925 22:48:20.387466 (+ 284us) negotiation.cc:107] Waiting for socket to connect
0925 22:48:20.387530 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.198.62:41557: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":202,"thread_start_us":81,"threads_started":1}
I20260925 22:48:20.614794 27973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:48:20.614309 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.198.62:41557 (local address 127.10.198.3:41713)
0925 22:48:20.614601 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0925 22:48:20.614664 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.198.62:41557: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":199,"thread_start_us":114,"threads_started":1}
I20260925 22:48:20.874672 27974 raft_consensus.cc:482] T 44c9a15c2ed944aebff7390622aa8768 P 84c93dfd42c94d188c85a636289a4e23 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 22:48:20.874953 27974 raft_consensus.cc:504] T 44c9a15c2ed944aebff7390622aa8768 P 84c93dfd42c94d188c85a636289a4e23 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e299c7a0d9b04d1a8d30de149122fc48" member_type: VOTER last_known_addr { host: "127.10.198.3" port: 43405 } } peers { permanent_uuid: "1b6eadd191934a76ba315f4c414dd647" member_type: VOTER last_known_addr { host: "127.10.198.2" port: 40709 } } peers { permanent_uuid: "84c93dfd42c94d188c85a636289a4e23" member_type: VOTER last_known_addr { host: "127.10.198.1" port: 36769 } }
I20260925 22:48:20.876168 27974 leader_election.cc:288] T 44c9a15c2ed944aebff7390622aa8768 P 84c93dfd42c94d188c85a636289a4e23 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e299c7a0d9b04d1a8d30de149122fc48 (127.10.198.3:43405), 1b6eadd191934a76ba315f4c414dd647 (127.10.198.2:40709)
Sep 25 22:48:20 dist-test-slave-5vdr krb5kdc[26391](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376499, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.198.1@KRBTEST.COM for kudu/127.10.198.3@KRBTEST.COM
Sep 25 22:48:20 dist-test-slave-5vdr krb5kdc[26391](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376499, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.198.1@KRBTEST.COM for kudu/127.10.198.2@KRBTEST.COM
I20260925 22:48:20.899230 27977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:48:20.876898 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.198.2:40709 (local address 127.10.198.1:54463)
0925 22:48:20.877175 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0925 22:48:20.877192 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 22:48:20.877305 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:48:20.879223 (+ 1918us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:48:20.879231 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:48:20.879893 (+ 662us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:48:20.880782 (+ 889us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:48:20.880808 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:48:20.883129 (+ 2321us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:48:20.883132 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:48:20.884615 (+ 1483us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:48:20.884632 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:48:20.885835 (+ 1203us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:48:20.885839 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:48:20.886171 (+ 332us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:48:20.890178 (+ 4007us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:48:20.890214 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:48:20.892873 (+ 2659us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:48:20.895507 (+ 2634us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:48:20.895510 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:48:20.895519 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:48:20.896161 (+ 642us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:48:20.896713 (+ 552us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:48:20.896716 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:48:20.896718 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:48:20.897007 (+ 289us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:48:20.897917 (+ 910us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:48:20.897925 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:48:20.898341 (+ 416us) client_negotiation.cc:769] Sending connection context
0925 22:48:20.898575 (+ 234us) client_negotiation.cc:240] Negotiation successful
0925 22:48:20.898984 (+ 409us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":207,"mutex_wait_us":373,"spinlock_wait_cycles":70912,"thread_start_us":48,"threads_started":1}
I20260925 22:48:20.899281 27978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:48:20.877055 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.198.1:54463 (local address 127.10.198.2:40709)
0925 22:48:20.877397 (+ 342us) server_negotiation.cc:206] Beginning negotiation
0925 22:48:20.877419 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0925 22:48:20.877455 (+ 36us) server_negotiation.cc:407] Connection header received
0925 22:48:20.878214 (+ 759us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:48:20.878222 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:48:20.878708 (+ 486us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:48:20.879083 (+ 375us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:48:20.881013 (+ 1930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:48:20.882630 (+ 1617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:48:20.884802 (+ 2172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:48:20.885652 (+ 850us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:48:20.885869 (+ 217us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:48:20.893121 (+ 7252us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:48:20.893128 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:48:20.893140 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:48:20.893175 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:48:20.895318 (+ 2143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:48:20.896343 (+ 1025us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:48:20.896351 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:48:20.896358 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:48:20.896590 (+ 232us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:48:20.897136 (+ 546us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:48:20.897140 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:48:20.897141 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:48:20.897753 (+ 612us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:48:20.897907 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0925 22:48:20.898747 (+ 840us) server_negotiation.cc:299] Negotiation successful
0925 22:48:20.899077 (+ 330us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":221,"thread_start_us":127,"threads_started":1}
I20260925 22:48:20.899287 27976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:48:20.876613 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.198.1:35059 (local address 127.10.198.3:43405)
0925 22:48:20.876967 (+ 354us) server_negotiation.cc:206] Beginning negotiation
0925 22:48:20.876999 (+ 32us) server_negotiation.cc:399] Waiting for connection header
0925 22:48:20.877148 (+ 149us) server_negotiation.cc:407] Connection header received
0925 22:48:20.878212 (+ 1064us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:48:20.878222 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:48:20.878707 (+ 485us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:48:20.879083 (+ 376us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:48:20.881013 (+ 1930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:48:20.882630 (+ 1617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:48:20.884802 (+ 2172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:48:20.885653 (+ 851us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:48:20.885878 (+ 225us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:48:20.892997 (+ 7119us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:48:20.893006 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:48:20.893032 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:48:20.893071 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:48:20.895308 (+ 2237us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:48:20.896343 (+ 1035us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:48:20.896351 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:48:20.896358 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:48:20.896590 (+ 232us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:48:20.897136 (+ 546us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:48:20.897140 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:48:20.897141 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:48:20.897753 (+ 612us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:48:20.897907 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0925 22:48:20.898747 (+ 840us) server_negotiation.cc:299] Negotiation successful
0925 22:48:20.899077 (+ 330us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":206,"thread_start_us":85,"threads_started":1}
I20260925 22:48:20.899230 27975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:48:20.876442 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.198.3:43405 (local address 127.10.198.1:35059)
0925 22:48:20.877025 (+ 583us) negotiation.cc:107] Waiting for socket to connect
0925 22:48:20.877041 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 22:48:20.877167 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:48:20.879223 (+ 2056us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:48:20.879231 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:48:20.879893 (+ 662us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:48:20.880782 (+ 889us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:48:20.880808 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:48:20.882844 (+ 2036us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:48:20.882848 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:48:20.884615 (+ 1767us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:48:20.884632 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:48:20.885835 (+ 1203us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:48:20.885839 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:48:20.886198 (+ 359us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:48:20.890178 (+ 3980us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:48:20.890230 (+ 52us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:48:20.892591 (+ 2361us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:48:20.895492 (+ 2901us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:48:20.895501 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:48:20.895519 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:48:20.896161 (+ 642us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:48:20.896713 (+ 552us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:48:20.896716 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:48:20.896718 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:48:20.897007 (+ 289us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:48:20.897917 (+ 910us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:48:20.897925 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:48:20.898341 (+ 416us) client_negotiation.cc:769] Sending connection context
0925 22:48:20.898575 (+ 234us) client_negotiation.cc:240] Negotiation successful
0925 22:48:20.899036 (+ 461us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":510,"mutex_wait_us":525,"spinlock_wait_cycles":81024,"thread_start_us":106,"threads_started":1}
W20260925 22:48:20.903319 27587 leader_election.cc:341] T 44c9a15c2ed944aebff7390622aa8768 P 84c93dfd42c94d188c85a636289a4e23 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer e299c7a0d9b04d1a8d30de149122fc48 (127.10.198.3:43405): Not found: Tablet not found: 44c9a15c2ed944aebff7390622aa8768
W20260925 22:48:20.903604 27587 leader_election.cc:341] T 44c9a15c2ed944aebff7390622aa8768 P 84c93dfd42c94d188c85a636289a4e23 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 1b6eadd191934a76ba315f4c414dd647 (127.10.198.2:40709): Not found: Tablet not found: 44c9a15c2ed944aebff7390622aa8768
I20260925 22:48:20.903689 27587 leader_election.cc:302] T 44c9a15c2ed944aebff7390622aa8768 P 84c93dfd42c94d188c85a636289a4e23 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 84c93dfd42c94d188c85a636289a4e23; no voters: 1b6eadd191934a76ba315f4c414dd647, e299c7a0d9b04d1a8d30de149122fc48
I20260925 22:48:20.904029 27974 raft_consensus.cc:2726] T 44c9a15c2ed944aebff7390622aa8768 P 84c93dfd42c94d188c85a636289a4e23 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 22:48:21.216482 27977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:48:21.216122 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.198.62:41557 (local address 127.10.198.1:53919)
0925 22:48:21.216310 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0925 22:48:21.216389 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.198.62:41557: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
I20260925 22:48:21.389029 27979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:48:21.388508 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.198.62:41557 (local address 127.10.198.2:51841)
0925 22:48:21.388834 (+ 326us) negotiation.cc:107] Waiting for socket to connect
0925 22:48:21.388907 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.198.62:41557: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":237,"thread_start_us":95,"threads_started":1}
I20260925 22:48:21.616338 27980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:48:21.615791 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.198.62:41557 (local address 127.10.198.3:48337)
0925 22:48:21.616143 (+ 352us) negotiation.cc:107] Waiting for socket to connect
0925 22:48:21.616210 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.198.62:41557: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":271,"thread_start_us":145,"threads_started":1}
I20260925 22:48:22.217958 27981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:48:22.217462 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.198.62:41557 (local address 127.10.198.1:34377)
0925 22:48:22.217783 (+ 321us) negotiation.cc:107] Waiting for socket to connect
0925 22:48:22.217857 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.198.62:41557: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":215,"thread_start_us":97,"threads_started":1}
I20260925 22:48:22.390478 27982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:48:22.389990 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.198.62:41557 (local address 127.10.198.2:59387)
0925 22:48:22.390290 (+ 300us) negotiation.cc:107] Waiting for socket to connect
0925 22:48:22.390373 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.198.62:41557: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":209,"thread_start_us":79,"threads_started":1}
I20260925 22:48:22.554566 27983 raft_consensus.cc:482] T 44c9a15c2ed944aebff7390622aa8768 P 84c93dfd42c94d188c85a636289a4e23 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 22:48:22.554733 27983 raft_consensus.cc:504] T 44c9a15c2ed944aebff7390622aa8768 P 84c93dfd42c94d188c85a636289a4e23 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e299c7a0d9b04d1a8d30de149122fc48" member_type: VOTER last_known_addr { host: "127.10.198.3" port: 43405 } } peers { permanent_uuid: "1b6eadd191934a76ba315f4c414dd647" member_type: VOTER last_known_addr { host: "127.10.198.2" port: 40709 } } peers { permanent_uuid: "84c93dfd42c94d188c85a636289a4e23" member_type: VOTER last_known_addr { host: "127.10.198.1" port: 36769 } }
I20260925 22:48:22.555411 27983 leader_election.cc:288] T 44c9a15c2ed944aebff7390622aa8768 P 84c93dfd42c94d188c85a636289a4e23 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e299c7a0d9b04d1a8d30de149122fc48 (127.10.198.3:43405), 1b6eadd191934a76ba315f4c414dd647 (127.10.198.2:40709)
W20260925 22:48:22.556353 27587 leader_election.cc:341] T 44c9a15c2ed944aebff7390622aa8768 P 84c93dfd42c94d188c85a636289a4e23 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer e299c7a0d9b04d1a8d30de149122fc48 (127.10.198.3:43405): Not found: Tablet not found: 44c9a15c2ed944aebff7390622aa8768
W20260925 22:48:22.556541 27587 leader_election.cc:341] T 44c9a15c2ed944aebff7390622aa8768 P 84c93dfd42c94d188c85a636289a4e23 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 1b6eadd191934a76ba315f4c414dd647 (127.10.198.2:40709): Not found: Tablet not found: 44c9a15c2ed944aebff7390622aa8768
I20260925 22:48:22.556607 27587 leader_election.cc:302] T 44c9a15c2ed944aebff7390622aa8768 P 84c93dfd42c94d188c85a636289a4e23 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 84c93dfd42c94d188c85a636289a4e23; no voters: 1b6eadd191934a76ba315f4c414dd647, e299c7a0d9b04d1a8d30de149122fc48
I20260925 22:48:22.556772 27983 raft_consensus.cc:2726] T 44c9a15c2ed944aebff7390622aa8768 P 84c93dfd42c94d188c85a636289a4e23 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 22:48:22.618162 27984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:48:22.617343 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.198.62:41557 (local address 127.10.198.3:51485)
0925 22:48:22.617705 (+ 362us) negotiation.cc:107] Waiting for socket to connect
0925 22:48:22.617961 (+ 256us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.198.62:41557: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":277,"thread_start_us":106,"threads_started":1}
I20260925 22:48:23.219496 27985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:48:23.219015 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.198.62:41557 (local address 127.10.198.1:47551)
0925 22:48:23.219324 (+ 309us) negotiation.cc:107] Waiting for socket to connect
0925 22:48:23.219396 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.198.62:41557: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":222,"thread_start_us":82,"threads_started":1}
I20260925 22:48:23.391775 27986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:48:23.391347 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.198.62:41557 (local address 127.10.198.2:36035)
0925 22:48:23.391600 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0925 22:48:23.391662 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.198.62:41557: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":78,"threads_started":1}
I20260925 22:48:23.619655 27987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:48:23.619165 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.198.62:41557 (local address 127.10.198.3:53723)
0925 22:48:23.619475 (+ 310us) negotiation.cc:107] Waiting for socket to connect
0925 22:48:23.619550 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.198.62:41557: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":223,"thread_start_us":87,"threads_started":1}
I20260925 22:48:24.221131 27988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:48:24.220636 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.198.62:41557 (local address 127.10.198.1:59969)
0925 22:48:24.220963 (+ 327us) negotiation.cc:107] Waiting for socket to connect
0925 22:48:24.221032 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.198.62:41557: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":242,"thread_start_us":82,"threads_started":1}
I20260925 22:48:24.393092 27989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:48:24.392657 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.198.62:41557 (local address 127.10.198.2:35335)
0925 22:48:24.392929 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0925 22:48:24.392997 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.198.62:41557: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":83,"threads_started":1}
I20260925 22:48:24.621044 27990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:48:24.620591 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.198.62:41557 (local address 127.10.198.3:52395)
0925 22:48:24.620866 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0925 22:48:24.620938 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.198.62:41557: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":81,"threads_started":1}
I20260925 22:48:25.003820 27991 raft_consensus.cc:482] T 44c9a15c2ed944aebff7390622aa8768 P 84c93dfd42c94d188c85a636289a4e23 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 22:48:25.004057 27991 raft_consensus.cc:504] T 44c9a15c2ed944aebff7390622aa8768 P 84c93dfd42c94d188c85a636289a4e23 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e299c7a0d9b04d1a8d30de149122fc48" member_type: VOTER last_known_addr { host: "127.10.198.3" port: 43405 } } peers { permanent_uuid: "1b6eadd191934a76ba315f4c414dd647" member_type: VOTER last_known_addr { host: "127.10.198.2" port: 40709 } } peers { permanent_uuid: "84c93dfd42c94d188c85a636289a4e23" member_type: VOTER last_known_addr { host: "127.10.198.1" port: 36769 } }
I20260925 22:48:25.004492 27991 leader_election.cc:288] T 44c9a15c2ed944aebff7390622aa8768 P 84c93dfd42c94d188c85a636289a4e23 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e299c7a0d9b04d1a8d30de149122fc48 (127.10.198.3:43405), 1b6eadd191934a76ba315f4c414dd647 (127.10.198.2:40709)
W20260925 22:48:25.005369 27587 leader_election.cc:341] T 44c9a15c2ed944aebff7390622aa8768 P 84c93dfd42c94d188c85a636289a4e23 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer e299c7a0d9b04d1a8d30de149122fc48 (127.10.198.3:43405): Not found: Tablet not found: 44c9a15c2ed944aebff7390622aa8768
W20260925 22:48:25.005533 27587 leader_election.cc:341] T 44c9a15c2ed944aebff7390622aa8768 P 84c93dfd42c94d188c85a636289a4e23 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 1b6eadd191934a76ba315f4c414dd647 (127.10.198.2:40709): Not found: Tablet not found: 44c9a15c2ed944aebff7390622aa8768
I20260925 22:48:25.005578 27587 leader_election.cc:302] T 44c9a15c2ed944aebff7390622aa8768 P 84c93dfd42c94d188c85a636289a4e23 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 84c93dfd42c94d188c85a636289a4e23; no voters: 1b6eadd191934a76ba315f4c414dd647, e299c7a0d9b04d1a8d30de149122fc48
I20260925 22:48:25.005736 27991 raft_consensus.cc:2726] T 44c9a15c2ed944aebff7390622aa8768 P 84c93dfd42c94d188c85a636289a4e23 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 22:48:25.222563 27992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:48:25.222066 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.198.62:41557 (local address 127.10.198.1:38543)
0925 22:48:25.222389 (+ 323us) negotiation.cc:107] Waiting for socket to connect
0925 22:48:25.222459 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.198.62:41557: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":233,"thread_start_us":84,"threads_started":1}
I20260925 22:48:25.394511 27993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:48:25.394054 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.198.62:41557 (local address 127.10.198.2:50579)
0925 22:48:25.394348 (+ 294us) negotiation.cc:107] Waiting for socket to connect
0925 22:48:25.394411 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.198.62:41557: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":215,"thread_start_us":84,"threads_started":1}
I20260925 22:48:25.622427 27994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:48:25.621957 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.198.62:41557 (local address 127.10.198.3:35819)
0925 22:48:25.622250 (+ 293us) negotiation.cc:107] Waiting for socket to connect
0925 22:48:25.622328 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.198.62:41557: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":209,"thread_start_us":89,"threads_started":1}
I20260925 22:48:26.224044 27995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:48:26.223491 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.198.62:41557 (local address 127.10.198.1:39193)
0925 22:48:26.223796 (+ 305us) negotiation.cc:107] Waiting for socket to connect
0925 22:48:26.223941 (+ 145us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.198.62:41557: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":217,"thread_start_us":98,"threads_started":1}
I20260925 22:48:26.395961 27996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:48:26.395326 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.198.62:41557 (local address 127.10.198.2:54327)
0925 22:48:26.395677 (+ 351us) negotiation.cc:107] Waiting for socket to connect
0925 22:48:26.395752 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.198.62:41557: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":265,"thread_start_us":82,"threads_started":1}
I20260925 22:48:26.624166 27997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:48:26.623450 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.198.62:41557 (local address 127.10.198.3:53251)
0925 22:48:26.623974 (+ 524us) negotiation.cc:107] Waiting for socket to connect
0925 22:48:26.624062 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.198.62:41557: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":314,"thread_start_us":86,"threads_started":1}
I20260925 22:48:27.225476 27998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:48:27.224949 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.198.62:41557 (local address 127.10.198.1:57809)
0925 22:48:27.225291 (+ 342us) negotiation.cc:107] Waiting for socket to connect
0925 22:48:27.225366 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.198.62:41557: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":245,"thread_start_us":80,"threads_started":1}
I20260925 22:48:27.397418 27999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:48:27.396905 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.198.62:41557 (local address 127.10.198.2:52817)
0925 22:48:27.397230 (+ 325us) negotiation.cc:107] Waiting for socket to connect
0925 22:48:27.397315 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.198.62:41557: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":226,"thread_start_us":89,"threads_started":1}
I20260925 22:48:27.625747 28000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:48:27.625265 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.198.62:41557 (local address 127.10.198.3:58623)
0925 22:48:27.625561 (+ 296us) negotiation.cc:107] Waiting for socket to connect
0925 22:48:27.625630 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.198.62:41557: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":214,"thread_start_us":90,"threads_started":1}
I20260925 22:48:27.714833 28001 raft_consensus.cc:482] T 44c9a15c2ed944aebff7390622aa8768 P 84c93dfd42c94d188c85a636289a4e23 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 22:48:27.715014 28001 raft_consensus.cc:504] T 44c9a15c2ed944aebff7390622aa8768 P 84c93dfd42c94d188c85a636289a4e23 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e299c7a0d9b04d1a8d30de149122fc48" member_type: VOTER last_known_addr { host: "127.10.198.3" port: 43405 } } peers { permanent_uuid: "1b6eadd191934a76ba315f4c414dd647" member_type: VOTER last_known_addr { host: "127.10.198.2" port: 40709 } } peers { permanent_uuid: "84c93dfd42c94d188c85a636289a4e23" member_type: VOTER last_known_addr { host: "127.10.198.1" port: 36769 } }
I20260925 22:48:27.715479 28001 leader_election.cc:288] T 44c9a15c2ed944aebff7390622aa8768 P 84c93dfd42c94d188c85a636289a4e23 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e299c7a0d9b04d1a8d30de149122fc48 (127.10.198.3:43405), 1b6eadd191934a76ba315f4c414dd647 (127.10.198.2:40709)
W20260925 22:48:27.716424 27587 leader_election.cc:341] T 44c9a15c2ed944aebff7390622aa8768 P 84c93dfd42c94d188c85a636289a4e23 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer e299c7a0d9b04d1a8d30de149122fc48 (127.10.198.3:43405): Not found: Tablet not found: 44c9a15c2ed944aebff7390622aa8768
W20260925 22:48:27.716593 27587 leader_election.cc:341] T 44c9a15c2ed944aebff7390622aa8768 P 84c93dfd42c94d188c85a636289a4e23 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 1b6eadd191934a76ba315f4c414dd647 (127.10.198.2:40709): Not found: Tablet not found: 44c9a15c2ed944aebff7390622aa8768
I20260925 22:48:27.716652 27587 leader_election.cc:302] T 44c9a15c2ed944aebff7390622aa8768 P 84c93dfd42c94d188c85a636289a4e23 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 84c93dfd42c94d188c85a636289a4e23; no voters: 1b6eadd191934a76ba315f4c414dd647, e299c7a0d9b04d1a8d30de149122fc48
I20260925 22:48:27.716864 28001 raft_consensus.cc:2726] T 44c9a15c2ed944aebff7390622aa8768 P 84c93dfd42c94d188c85a636289a4e23 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 22:48:28.227150 28002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:48:28.226610 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.198.62:41557 (local address 127.10.198.1:44529)
0925 22:48:28.226957 (+ 347us) negotiation.cc:107] Waiting for socket to connect
0925 22:48:28.227041 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.198.62:41557: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":260,"thread_start_us":105,"threads_started":1}
I20260925 22:48:28.398758 28003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:48:28.398321 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.198.62:41557 (local address 127.10.198.2:44799)
0925 22:48:28.398594 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0925 22:48:28.398658 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.198.62:41557: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":93,"threads_started":1}
I20260925 22:48:28.627305 28004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:48:28.626796 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.198.62:41557 (local address 127.10.198.3:40635)
0925 22:48:28.627110 (+ 314us) negotiation.cc:107] Waiting for socket to connect
0925 22:48:28.627194 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.198.62:41557: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":231,"thread_start_us":101,"threads_started":1}
I20260925 22:48:29.228840 28005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:48:29.228339 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.198.62:41557 (local address 127.10.198.1:33737)
0925 22:48:29.228651 (+ 312us) negotiation.cc:107] Waiting for socket to connect
0925 22:48:29.228736 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.198.62:41557: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":223,"thread_start_us":87,"threads_started":1}
I20260925 22:48:29.400107 28006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:48:29.399638 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.198.62:41557 (local address 127.10.198.2:37133)
0925 22:48:29.399947 (+ 309us) negotiation.cc:107] Waiting for socket to connect
0925 22:48:29.400010 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.198.62:41557: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":224,"thread_start_us":120,"threads_started":1}
I20260925 22:48:29.628805 28007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:48:29.628323 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.198.62:41557 (local address 127.10.198.3:46851)
0925 22:48:29.628631 (+ 308us) negotiation.cc:107] Waiting for socket to connect
0925 22:48:29.628702 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.198.62:41557: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":226,"thread_start_us":82,"threads_started":1}
I20260925 22:48:30.230491 28008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:48:30.229902 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.198.62:41557 (local address 127.10.198.1:42511)
0925 22:48:30.230293 (+ 391us) negotiation.cc:107] Waiting for socket to connect
0925 22:48:30.230387 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.198.62:41557: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":278,"thread_start_us":105,"threads_started":1}
I20260925 22:48:30.401500 28009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:48:30.400960 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.198.62:41557 (local address 127.10.198.2:48829)
0925 22:48:30.401305 (+ 345us) negotiation.cc:107] Waiting for socket to connect
0925 22:48:30.401390 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.198.62:41557: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":256,"thread_start_us":95,"threads_started":1}
I20260925 22:48:30.630353 28010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:48:30.629823 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.198.62:41557 (local address 127.10.198.3:45039)
0925 22:48:30.630142 (+ 319us) negotiation.cc:107] Waiting for socket to connect
0925 22:48:30.630225 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.198.62:41557: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":245,"thread_start_us":129,"threads_started":1}
I20260925 22:48:31.231897 28011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:48:31.231355 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.198.62:41557 (local address 127.10.198.1:42109)
0925 22:48:31.231692 (+ 337us) negotiation.cc:107] Waiting for socket to connect
0925 22:48:31.231760 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.198.62:41557: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":234,"thread_start_us":99,"threads_started":1}
I20260925 22:48:31.403146 28012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:48:31.402663 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.198.62:41557 (local address 127.10.198.2:43809)
0925 22:48:31.402980 (+ 317us) negotiation.cc:107] Waiting for socket to connect
0925 22:48:31.403050 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.198.62:41557: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":223,"thread_start_us":103,"threads_started":1}
I20260925 22:48:31.631608 28013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:48:31.631165 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.198.62:41557 (local address 127.10.198.3:56681)
0925 22:48:31.631451 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0925 22:48:31.631512 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.198.62:41557: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":214,"thread_start_us":94,"threads_started":1}
I20260925 22:48:32.233299 28014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:48:32.232774 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.198.62:41557 (local address 127.10.198.1:39203)
0925 22:48:32.233111 (+ 337us) negotiation.cc:107] Waiting for socket to connect
0925 22:48:32.233182 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.198.62:41557: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":238,"thread_start_us":99,"threads_started":1}
I20260925 22:48:32.404769 28015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:48:32.404305 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.198.62:41557 (local address 127.10.198.2:38123)
0925 22:48:32.404595 (+ 290us) negotiation.cc:107] Waiting for socket to connect
0925 22:48:32.404664 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.198.62:41557: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":202,"thread_start_us":79,"threads_started":1}
I20260925 22:48:32.632910 28016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:48:32.632445 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.198.62:41557 (local address 127.10.198.3:57799)
0925 22:48:32.632745 (+ 300us) negotiation.cc:107] Waiting for socket to connect
0925 22:48:32.632810 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.198.62:41557: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":214,"thread_start_us":90,"threads_started":1}
I20260925 22:48:32.859827 28017 raft_consensus.cc:482] T 44c9a15c2ed944aebff7390622aa8768 P 84c93dfd42c94d188c85a636289a4e23 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 22:48:32.860073 28017 raft_consensus.cc:504] T 44c9a15c2ed944aebff7390622aa8768 P 84c93dfd42c94d188c85a636289a4e23 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e299c7a0d9b04d1a8d30de149122fc48" member_type: VOTER last_known_addr { host: "127.10.198.3" port: 43405 } } peers { permanent_uuid: "1b6eadd191934a76ba315f4c414dd647" member_type: VOTER last_known_addr { host: "127.10.198.2" port: 40709 } } peers { permanent_uuid: "84c93dfd42c94d188c85a636289a4e23" member_type: VOTER last_known_addr { host: "127.10.198.1" port: 36769 } }
I20260925 22:48:32.860628 28017 leader_election.cc:288] T 44c9a15c2ed944aebff7390622aa8768 P 84c93dfd42c94d188c85a636289a4e23 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e299c7a0d9b04d1a8d30de149122fc48 (127.10.198.3:43405), 1b6eadd191934a76ba315f4c414dd647 (127.10.198.2:40709)
W20260925 22:48:32.861554 27587 leader_election.cc:341] T 44c9a15c2ed944aebff7390622aa8768 P 84c93dfd42c94d188c85a636289a4e23 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer e299c7a0d9b04d1a8d30de149122fc48 (127.10.198.3:43405): Not found: Tablet not found: 44c9a15c2ed944aebff7390622aa8768
W20260925 22:48:32.861764 27587 leader_election.cc:341] T 44c9a15c2ed944aebff7390622aa8768 P 84c93dfd42c94d188c85a636289a4e23 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 1b6eadd191934a76ba315f4c414dd647 (127.10.198.2:40709): Not found: Tablet not found: 44c9a15c2ed944aebff7390622aa8768
I20260925 22:48:32.861827 27587 leader_election.cc:302] T 44c9a15c2ed944aebff7390622aa8768 P 84c93dfd42c94d188c85a636289a4e23 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 84c93dfd42c94d188c85a636289a4e23; no voters: 1b6eadd191934a76ba315f4c414dd647, e299c7a0d9b04d1a8d30de149122fc48
I20260925 22:48:32.862027 28017 raft_consensus.cc:2726] T 44c9a15c2ed944aebff7390622aa8768 P 84c93dfd42c94d188c85a636289a4e23 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 22:48:33.234627 28018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:48:33.234196 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.198.62:41557 (local address 127.10.198.1:56599)
0925 22:48:33.234464 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0925 22:48:33.234526 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.198.62:41557: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":79,"threads_started":1}
I20260925 22:48:33.406284 28019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:48:33.405761 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.198.62:41557 (local address 127.10.198.2:59023)
0925 22:48:33.406078 (+ 317us) negotiation.cc:107] Waiting for socket to connect
0925 22:48:33.406182 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.198.62:41557: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":228,"thread_start_us":93,"threads_started":1}
I20260925 22:48:33.634622 28020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:48:33.634137 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.198.62:41557 (local address 127.10.198.3:53409)
0925 22:48:33.634428 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0925 22:48:33.634489 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.198.62:41557: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":104,"threads_started":1}
I20260925 22:48:34.236414 28021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:48:34.235750 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.198.62:41557 (local address 127.10.198.1:51445)
0925 22:48:34.236237 (+ 487us) negotiation.cc:107] Waiting for socket to connect
0925 22:48:34.236309 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.198.62:41557: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":388,"thread_start_us":128,"threads_started":1}
I20260925 22:48:34.408006 28022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:48:34.407375 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.198.62:41557 (local address 127.10.198.2:42333)
0925 22:48:34.407751 (+ 376us) negotiation.cc:107] Waiting for socket to connect
0925 22:48:34.407823 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.198.62:41557: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":287,"thread_start_us":79,"threads_started":1}
I20260925 22:48:34.622174 11032 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrWXm8X/build/debug/bin/kudu with pid 27538
I20260925 22:48:34.628407 11032 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrWXm8X/build/debug/bin/kudu with pid 27704
I20260925 22:48:34.634333 11032 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrWXm8X/build/debug/bin/kudu with pid 27838
2026-09-25T22:48:34Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (59588 ms)
[----------] 30 tests from SecurityITest (254973 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790376259650642-11032-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 22:48:34 dist-test-slave-5vdr krb5kdc[28025](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 22:48:34 dist-test-slave-5vdr krb5kdc[28025](info): set up 2 sockets
Sep 25 22:48:34 dist-test-slave-5vdr krb5kdc[28025](info): commencing operation
krb5kdc: starting...
W20260925 22:48:35.695806 11032 mini_kdc.cc:121] Time spent starting KDC: real 1.038s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790376259650642-11032-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790376259650642-11032-0/krb5kdc/test-user.keytab.
Sep 25 22:48:35 dist-test-slave-5vdr krb5kdc[28025](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376515, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T22:48:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T22:48:35Z Disabled control of system clock
WARNING: no policy specified for kudu/127.10.198.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.198.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.198.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790376259650642-11032-0/krb5kdc/kudu_127.10.198.62.keytab.
Entry for principal kudu/127.10.198.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790376259650642-11032-0/krb5kdc/kudu_127.10.198.62.keytab.
I20260925 22:48:35.831439 11032 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrWXm8X/build/debug/bin/kudu
/tmp/dist-test-taskrWXm8X/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790376259650642-11032-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790376259650642-11032-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790376259650642-11032-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790376259650642-11032-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.198.62:33175
--webserver_interface=127.10.198.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790376259650642-11032-0/krb5kdc/kudu_127.10.198.62.keytab
--principal=kudu/127.10.198.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.198.20:37837
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.198.62:33175
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790376259650642-11032-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790376259650642-11032-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790376259650642-11032-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:48:35.948086 28038 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:48:35.948408 28038 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:48:35.948454 28038 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:48:35.952775 28038 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 22:48:35.952865 28038 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:48:35.952885 28038 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:48:35.952903 28038 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 22:48:35.952919 28038 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 22:48:35.958693 28038 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.198.20:37837
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790376259650642-11032-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790376259650642-11032-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.198.62:33175
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790376259650642-11032-0/krb5kdc/kudu_127.10.198.62.keytab
--principal=kudu/127.10.198.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.198.62:33175
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790376259650642-11032-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.198.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28038
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790376259650642-11032-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:09:16 UTC on ccycloud.quasar-afmhyw.root.comops.site
build id 4673379
I20260925 22:48:35.959923 28038 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:48:35.961054 28038 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 22:48:35.970983 28038 server_base.cc:1034] running on GCE node
W20260925 22:48:35.970918 28043 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:48:35.970918 28046 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:48:35.970920 28044 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:48:35.972033 28038 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:48:35.973368 28038 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:48:35.974607 28038 hybrid_clock.cc:648] HybridClock initialized: now 1790376515974569 us; error 55 us; skew 500 ppm
Sep 25 22:48:35 dist-test-slave-5vdr krb5kdc[28025](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376515, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.198.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:48:35.977926 28038 init.cc:377] Logged in from keytab as kudu/127.10.198.62@KRBTEST.COM (short username kudu)
I20260925 22:48:35.979066 28038 webserver.cc:460] Webserver started at http://127.10.198.62:37475/ using document root <none> and password file <none>
I20260925 22:48:35.979722 28038 fs_manager.cc:359] Metadata directory not provided
I20260925 22:48:35.979827 28038 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:48:35.980129 28038 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:48:35.982452 28038 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790376259650642-11032-0/minicluster-data/master-0/data/instance:
uuid: "e124597a34ed4eaea13ca5444fc3cea8"
format_stamp: "Formatted at 2026-09-25 22:48:35 on dist-test-slave-5vdr"
server_key: "20629714925d6dd85c4c4777aa147dc7"
server_key_iv: "f21c5df116daf9700ec8f3f4c590dabf"
server_key_version: "encryptionkey@0"
I20260925 22:48:35.983115 28038 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790376259650642-11032-0/minicluster-data/master-0/wal/instance:
uuid: "e124597a34ed4eaea13ca5444fc3cea8"
format_stamp: "Formatted at 2026-09-25 22:48:35 on dist-test-slave-5vdr"
server_key: "20629714925d6dd85c4c4777aa147dc7"
server_key_iv: "f21c5df116daf9700ec8f3f4c590dabf"
server_key_version: "encryptionkey@0"
I20260925 22:48:35.987454 28038 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260925 22:48:35.990402 28053 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:48:35.991899 28038 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260925 22:48:35.992077 28038 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790376259650642-11032-0/minicluster-data/master-0/data,/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790376259650642-11032-0/minicluster-data/master-0/wal
uuid: "e124597a34ed4eaea13ca5444fc3cea8"
format_stamp: "Formatted at 2026-09-25 22:48:35 on dist-test-slave-5vdr"
server_key: "20629714925d6dd85c4c4777aa147dc7"
server_key_iv: "f21c5df116daf9700ec8f3f4c590dabf"
server_key_version: "encryptionkey@0"
I20260925 22:48:35.992218 28038 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790376259650642-11032-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790376259650642-11032-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790376259650642-11032-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:48:36.017293 28038 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:48:36.020687 28038 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:48:36.020934 28038 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:48:36.028852 28038 rpc_server.cc:307] RPC server started. Bound to: 127.10.198.62:33175
I20260925 22:48:36.028914 28105 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.198.62:33175 every 8 connection(s)
I20260925 22:48:36.029946 28038 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790376259650642-11032-0/minicluster-data/master-0/data/info.pb
I20260925 22:48:36.033036 28106 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:48:36.039208 28106 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e124597a34ed4eaea13ca5444fc3cea8: Bootstrap starting.
I20260925 22:48:36.039458 11032 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrWXm8X/build/debug/bin/kudu as pid 28038
I20260925 22:48:36.039606 11032 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790376259650642-11032-0/minicluster-data/master-0/wal/instance
I20260925 22:48:36.039973 11032 external_mini_cluster.cc:1428] Setting key 0a48bd3eb87747f276666d5d803e57ed
I20260925 22:48:36.042201 28106 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e124597a34ed4eaea13ca5444fc3cea8: Neither blocks nor log segments found. Creating new log.
I20260925 22:48:36.043017 28106 log.cc:824] T 00000000000000000000000000000000 P e124597a34ed4eaea13ca5444fc3cea8: Log is configured to *not* fsync() on all Append() calls
I20260925 22:48:36.045477 28106 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e124597a34ed4eaea13ca5444fc3cea8: No bootstrap required, opened a new log
Sep 25 22:48:36 dist-test-slave-5vdr krb5kdc[28025](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376515, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.198.62@KRBTEST.COM
I20260925 22:48:36.050544 28106 raft_consensus.cc:348] T 00000000000000000000000000000000 P e124597a34ed4eaea13ca5444fc3cea8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e124597a34ed4eaea13ca5444fc3cea8" member_type: VOTER last_known_addr { host: "127.10.198.62" port: 33175 } }
I20260925 22:48:36.050845 28106 raft_consensus.cc:374] T 00000000000000000000000000000000 P e124597a34ed4eaea13ca5444fc3cea8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:48:36.050910 28106 raft_consensus.cc:729] T 00000000000000000000000000000000 P e124597a34ed4eaea13ca5444fc3cea8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e124597a34ed4eaea13ca5444fc3cea8, State: Initialized, Role: FOLLOWER
I20260925 22:48:36.051481 28106 consensus_queue.cc:260] T 00000000000000000000000000000000 P e124597a34ed4eaea13ca5444fc3cea8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e124597a34ed4eaea13ca5444fc3cea8" member_type: VOTER last_known_addr { host: "127.10.198.62" port: 33175 } }
I20260925 22:48:36.051693 28106 raft_consensus.cc:388] T 00000000000000000000000000000000 P e124597a34ed4eaea13ca5444fc3cea8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 22:48:36.051802 28106 raft_consensus.cc:482] T 00000000000000000000000000000000 P e124597a34ed4eaea13ca5444fc3cea8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 22:48:36.051982 28106 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e124597a34ed4eaea13ca5444fc3cea8 [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:48:36.053010 28106 raft_consensus.cc:504] T 00000000000000000000000000000000 P e124597a34ed4eaea13ca5444fc3cea8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e124597a34ed4eaea13ca5444fc3cea8" member_type: VOTER last_known_addr { host: "127.10.198.62" port: 33175 } }
I20260925 22:48:36.053411 28106 leader_election.cc:302] T 00000000000000000000000000000000 P e124597a34ed4eaea13ca5444fc3cea8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e124597a34ed4eaea13ca5444fc3cea8; no voters:
I20260925 22:48:36.053793 28106 leader_election.cc:288] T 00000000000000000000000000000000 P e124597a34ed4eaea13ca5444fc3cea8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 22:48:36.053884 28111 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e124597a34ed4eaea13ca5444fc3cea8 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 22:48:36.054073 28111 raft_consensus.cc:686] T 00000000000000000000000000000000 P e124597a34ed4eaea13ca5444fc3cea8 [term 1 LEADER]: Becoming Leader. State: Replica: e124597a34ed4eaea13ca5444fc3cea8, State: Running, Role: LEADER
I20260925 22:48:36.054467 28111 consensus_queue.cc:237] T 00000000000000000000000000000000 P e124597a34ed4eaea13ca5444fc3cea8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e124597a34ed4eaea13ca5444fc3cea8" member_type: VOTER last_known_addr { host: "127.10.198.62" port: 33175 } }
I20260925 22:48:36.055264 28106 sys_catalog.cc:564] T 00000000000000000000000000000000 P e124597a34ed4eaea13ca5444fc3cea8 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 22:48:36.056322 28112 sys_catalog.cc:455] T 00000000000000000000000000000000 P e124597a34ed4eaea13ca5444fc3cea8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e124597a34ed4eaea13ca5444fc3cea8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e124597a34ed4eaea13ca5444fc3cea8" member_type: VOTER last_known_addr { host: "127.10.198.62" port: 33175 } } }
I20260925 22:48:36.056784 28109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:48:36.041459 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54174 (local address 127.10.198.62:33175)
0925 22:48:36.041921 (+ 462us) server_negotiation.cc:206] Beginning negotiation
0925 22:48:36.041939 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0925 22:48:36.041976 (+ 37us) server_negotiation.cc:407] Connection header received
0925 22:48:36.042743 (+ 767us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:48:36.042764 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:48:36.043133 (+ 369us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:48:36.043572 (+ 439us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:48:36.044519 (+ 947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:48:36.045544 (+ 1025us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:48:36.046775 (+ 1231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:48:36.047389 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:48:36.047530 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:48:36.050634 (+ 3104us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:48:36.050641 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:48:36.050659 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:48:36.050696 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:48:36.052932 (+ 2236us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:48:36.053530 (+ 598us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:48:36.053537 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:48:36.053544 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:48:36.053713 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:48:36.054079 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:48:36.054083 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:48:36.054085 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:48:36.054662 (+ 577us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:48:36.054912 (+ 250us) server_negotiation.cc:1035] Waiting for connection context
0925 22:48:36.055214 (+ 302us) server_negotiation.cc:299] Negotiation successful
0925 22:48:36.055576 (+ 362us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":301,"thread_start_us":158,"threads_started":1}
I20260925 22:48:36.056703 28113 sys_catalog.cc:455] T 00000000000000000000000000000000 P e124597a34ed4eaea13ca5444fc3cea8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e124597a34ed4eaea13ca5444fc3cea8. Latest consensus state: current_term: 1 leader_uuid: "e124597a34ed4eaea13ca5444fc3cea8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e124597a34ed4eaea13ca5444fc3cea8" member_type: VOTER last_known_addr { host: "127.10.198.62" port: 33175 } } }
I20260925 22:48:36.057010 28113 sys_catalog.cc:458] T 00000000000000000000000000000000 P e124597a34ed4eaea13ca5444fc3cea8 [sys.catalog]: This master's current role is: LEADER
I20260925 22:48:36.056824 28112 sys_catalog.cc:458] T 00000000000000000000000000000000 P e124597a34ed4eaea13ca5444fc3cea8 [sys.catalog]: This master's current role is: LEADER
I20260925 22:48:36.057467 28123 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 22:48:36.061106 28123 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 22:48:36.067199 28123 catalog_manager.cc:1347] Generated new cluster ID: d4376e209f8a4304a2957e8f1a0d103d
I20260925 22:48:36.067296 28123 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 22:48:36.083242 28123 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 22:48:36.084409 28123 catalog_manager.cc:1504] Loading token signing keys...
I20260925 22:48:36.089807 28123 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e124597a34ed4eaea13ca5444fc3cea8: Generated new TSK 0
I20260925 22:48:36.090762 28123 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.10.198.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.198.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.198.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790376259650642-11032-0/krb5kdc/kudu_127.10.198.1.keytab.
Entry for principal kudu/127.10.198.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790376259650642-11032-0/krb5kdc/kudu_127.10.198.1.keytab.
I20260925 22:48:36.123577 11032 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrWXm8X/build/debug/bin/kudu
/tmp/dist-test-taskrWXm8X/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790376259650642-11032-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790376259650642-11032-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790376259650642-11032-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790376259650642-11032-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.198.1:0
--local_ip_for_outbound_sockets=127.10.198.1
--webserver_interface=127.10.198.1
--webserver_port=0
--tserver_master_addrs=127.10.198.62:33175
--keytab_file=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790376259650642-11032-0/krb5kdc/kudu_127.10.198.1.keytab
--principal=kudu/127.10.198.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.198.20:37837
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790376259650642-11032-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790376259650642-11032-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790376259650642-11032-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:48:36.246973 28132 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:48:36.247308 28132 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:48:36.247354 28132 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:48:36.252462 28132 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:48:36.252557 28132 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:48:36.252657 28132 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.198.1
I20260925 22:48:36.258980 28132 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.198.20:37837
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790376259650642-11032-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790376259650642-11032-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790376259650642-11032-0/krb5kdc/kudu_127.10.198.1.keytab
--principal=kudu/127.10.198.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.198.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790376259650642-11032-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.198.1
--webserver_port=0
--tserver_master_addrs=127.10.198.62:33175
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28132
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.198.1
--log_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790376259650642-11032-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:09:16 UTC on ccycloud.quasar-afmhyw.root.comops.site
build id 4673379
I20260925 22:48:36.260336 28132 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:48:36.261475 28132 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:48:36.272459 28137 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:48:36.272652 28132 server_base.cc:1034] running on GCE node
W20260925 22:48:36.272461 28138 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:48:36.272832 28140 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:48:36.273267 28132 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:48:36.273962 28132 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:48:36.275164 28132 hybrid_clock.cc:648] HybridClock initialized: now 1790376516275137 us; error 48 us; skew 500 ppm
Sep 25 22:48:36 dist-test-slave-5vdr krb5kdc[28025](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376516, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.198.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:48:36.278230 28132 init.cc:377] Logged in from keytab as kudu/127.10.198.1@KRBTEST.COM (short username kudu)
I20260925 22:48:36.279233 28132 webserver.cc:460] Webserver started at http://127.10.198.1:36659/ using document root <none> and password file <none>
I20260925 22:48:36.279906 28132 fs_manager.cc:359] Metadata directory not provided
I20260925 22:48:36.280005 28132 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:48:36.280247 28132 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:48:36.282492 28132 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790376259650642-11032-0/minicluster-data/ts-0/data/instance:
uuid: "d812b13d48aa401e94c7572985e507f3"
format_stamp: "Formatted at 2026-09-25 22:48:36 on dist-test-slave-5vdr"
server_key: "727afcf6fd0133e03ded6dc95dd08512"
server_key_iv: "d48b5b64833305f0da8b93b1b36bcc26"
server_key_version: "encryptionkey@0"
I20260925 22:48:36.283147 28132 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790376259650642-11032-0/minicluster-data/ts-0/wal/instance:
uuid: "d812b13d48aa401e94c7572985e507f3"
format_stamp: "Formatted at 2026-09-25 22:48:36 on dist-test-slave-5vdr"
server_key: "727afcf6fd0133e03ded6dc95dd08512"
server_key_iv: "d48b5b64833305f0da8b93b1b36bcc26"
server_key_version: "encryptionkey@0"
I20260925 22:48:36.287593 28132 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260925 22:48:36.290673 28147 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:48:36.292156 28132 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 22:48:36.292337 28132 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790376259650642-11032-0/minicluster-data/ts-0/data,/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790376259650642-11032-0/minicluster-data/ts-0/wal
uuid: "d812b13d48aa401e94c7572985e507f3"
format_stamp: "Formatted at 2026-09-25 22:48:36 on dist-test-slave-5vdr"
server_key: "727afcf6fd0133e03ded6dc95dd08512"
server_key_iv: "d48b5b64833305f0da8b93b1b36bcc26"
server_key_version: "encryptionkey@0"
I20260925 22:48:36.292478 28132 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790376259650642-11032-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790376259650642-11032-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790376259650642-11032-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:48:36.307317 28132 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:48:36.310709 28132 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:48:36.310956 28132 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:48:36.311611 28132 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 22:48:36.312642 28132 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:48:36.312719 28132 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:48:36.312794 28132 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:48:36.312835 28132 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 22:48:36.323231 28132 rpc_server.cc:307] RPC server started. Bound to: 127.10.198.1:43261
I20260925 22:48:36.323316 28260 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.198.1:43261 every 8 connection(s)
I20260925 22:48:36.324347 28132 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790376259650642-11032-0/minicluster-data/ts-0/data/info.pb
I20260925 22:48:36.331584 11032 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrWXm8X/build/debug/bin/kudu as pid 28132
I20260925 22:48:36.331759 11032 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790376259650642-11032-0/minicluster-data/ts-0/wal/instance
I20260925 22:48:36.332166 11032 external_mini_cluster.cc:1428] Setting key 5850d6dcd72b19ca17c747e377faaf38
Sep 25 22:48:36 dist-test-slave-5vdr krb5kdc[28025](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376516, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.198.1@KRBTEST.COM for kudu/127.10.198.62@KRBTEST.COM
I20260925 22:48:36.340269 28109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:48:36.326413 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.198.1:49433 (local address 127.10.198.62:33175)
0925 22:48:36.326568 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0925 22:48:36.326572 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:48:36.327830 (+ 1258us) server_negotiation.cc:407] Connection header received
0925 22:48:36.328948 (+ 1118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:48:36.328951 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:48:36.329006 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:48:36.329077 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:48:36.330474 (+ 1397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:48:36.331008 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:48:36.332906 (+ 1898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:48:36.333158 (+ 252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:48:36.333305 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:48:36.336219 (+ 2914us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:48:36.336223 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:48:36.336225 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:48:36.336254 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:48:36.337787 (+ 1533us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:48:36.338446 (+ 659us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:48:36.338450 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:48:36.338451 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:48:36.338536 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:48:36.338937 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:48:36.338940 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:48:36.338946 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:48:36.339150 (+ 204us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:48:36.339263 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0925 22:48:36.339923 (+ 660us) server_negotiation.cc:299] Negotiation successful
0925 22:48:36.340081 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260925 22:48:36.341239 28263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:48:36.326843 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.198.62:33175 (local address 127.10.198.1:49433)
0925 22:48:36.327628 (+ 785us) negotiation.cc:107] Waiting for socket to connect
0925 22:48:36.327681 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0925 22:48:36.328691 (+ 1010us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:48:36.329284 (+ 593us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:48:36.329291 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:48:36.329740 (+ 449us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:48:36.330297 (+ 557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:48:36.330314 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:48:36.331162 (+ 848us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:48:36.331165 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:48:36.332730 (+ 1565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:48:36.332745 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:48:36.333347 (+ 602us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:48:36.333351 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:48:36.333547 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:48:36.334332 (+ 785us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:48:36.334351 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:48:36.335998 (+ 1647us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:48:36.337948 (+ 1950us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:48:36.337956 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:48:36.337976 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:48:36.338314 (+ 338us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:48:36.338637 (+ 323us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:48:36.338641 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:48:36.338643 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:48:36.338824 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:48:36.339265 (+ 441us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:48:36.339284 (+ 19us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:48:36.339587 (+ 303us) client_negotiation.cc:769] Sending connection context
0925 22:48:36.339795 (+ 208us) client_negotiation.cc:240] Negotiation successful
0925 22:48:36.340235 (+ 440us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":543,"thread_start_us":124,"threads_started":1}
WARNING: no policy specified for kudu/127.10.198.2@KRBTEST.COM; defaulting to no policy
I20260925 22:48:36.342949 28261 heartbeater.cc:347] Connected to a master server at 127.10.198.62:33175
I20260925 22:48:36.343268 28261 heartbeater.cc:464] Registering TS with master...
I20260925 22:48:36.343981 28261 heartbeater.cc:511] Master 127.10.198.62:33175 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.198.2@KRBTEST.COM" created.
I20260925 22:48:36.345877 28070 ts_manager.cc:194] Registered new tserver with Master: d812b13d48aa401e94c7572985e507f3 (127.10.198.1:43261)
I20260925 22:48:36.347534 28070 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.198.1@KRBTEST.COM'} at 127.10.198.1:49433
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.198.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790376259650642-11032-0/krb5kdc/kudu_127.10.198.2.keytab.
Entry for principal kudu/127.10.198.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790376259650642-11032-0/krb5kdc/kudu_127.10.198.2.keytab.
I20260925 22:48:36.363371 11032 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrWXm8X/build/debug/bin/kudu
/tmp/dist-test-taskrWXm8X/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790376259650642-11032-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790376259650642-11032-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790376259650642-11032-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790376259650642-11032-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.198.2:0
--local_ip_for_outbound_sockets=127.10.198.2
--webserver_interface=127.10.198.2
--webserver_port=0
--tserver_master_addrs=127.10.198.62:33175
--keytab_file=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790376259650642-11032-0/krb5kdc/kudu_127.10.198.2.keytab
--principal=kudu/127.10.198.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.198.20:37837
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790376259650642-11032-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790376259650642-11032-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790376259650642-11032-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:48:36.484275 28266 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:48:36.484591 28266 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:48:36.484637 28266 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:48:36.489537 28266 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:48:36.489622 28266 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:48:36.489692 28266 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.198.2
I20260925 22:48:36.495671 28266 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.198.20:37837
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790376259650642-11032-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790376259650642-11032-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790376259650642-11032-0/krb5kdc/kudu_127.10.198.2.keytab
--principal=kudu/127.10.198.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.198.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790376259650642-11032-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.198.2
--webserver_port=0
--tserver_master_addrs=127.10.198.62:33175
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28266
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.198.2
--log_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790376259650642-11032-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:09:16 UTC on ccycloud.quasar-afmhyw.root.comops.site
build id 4673379
I20260925 22:48:36.497025 28266 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:48:36.498198 28266 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:48:36.508929 28271 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:48:36.508936 28272 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:48:36.508936 28274 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:48:36.509178 28266 server_base.cc:1034] running on GCE node
I20260925 22:48:36.509775 28266 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:48:36.510525 28266 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:48:36.511750 28266 hybrid_clock.cc:648] HybridClock initialized: now 1790376516511713 us; error 58 us; skew 500 ppm
Sep 25 22:48:36 dist-test-slave-5vdr krb5kdc[28025](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376516, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.198.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:48:36.514868 28266 init.cc:377] Logged in from keytab as kudu/127.10.198.2@KRBTEST.COM (short username kudu)
I20260925 22:48:36.516043 28266 webserver.cc:460] Webserver started at http://127.10.198.2:34207/ using document root <none> and password file <none>
I20260925 22:48:36.516779 28266 fs_manager.cc:359] Metadata directory not provided
I20260925 22:48:36.516886 28266 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:48:36.517133 28266 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:48:36.519424 28266 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790376259650642-11032-0/minicluster-data/ts-1/data/instance:
uuid: "821cc55a7f9941c7949757511c90a2a7"
format_stamp: "Formatted at 2026-09-25 22:48:36 on dist-test-slave-5vdr"
server_key: "cf86ecf39a6e09799956cee5f0a0a3c3"
server_key_iv: "ebbacc7240c4d668284a12713216fec3"
server_key_version: "encryptionkey@0"
I20260925 22:48:36.520437 28266 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790376259650642-11032-0/minicluster-data/ts-1/wal/instance:
uuid: "821cc55a7f9941c7949757511c90a2a7"
format_stamp: "Formatted at 2026-09-25 22:48:36 on dist-test-slave-5vdr"
server_key: "cf86ecf39a6e09799956cee5f0a0a3c3"
server_key_iv: "ebbacc7240c4d668284a12713216fec3"
server_key_version: "encryptionkey@0"
I20260925 22:48:36.525161 28266 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260925 22:48:36.528168 28281 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:48:36.529570 28266 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260925 22:48:36.529750 28266 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790376259650642-11032-0/minicluster-data/ts-1/data,/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790376259650642-11032-0/minicluster-data/ts-1/wal
uuid: "821cc55a7f9941c7949757511c90a2a7"
format_stamp: "Formatted at 2026-09-25 22:48:36 on dist-test-slave-5vdr"
server_key: "cf86ecf39a6e09799956cee5f0a0a3c3"
server_key_iv: "ebbacc7240c4d668284a12713216fec3"
server_key_version: "encryptionkey@0"
I20260925 22:48:36.529902 28266 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790376259650642-11032-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790376259650642-11032-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790376259650642-11032-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:48:36.547640 28266 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:48:36.551146 28266 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:48:36.551369 28266 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:48:36.552068 28266 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 22:48:36.552977 28266 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:48:36.553050 28266 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:48:36.553121 28266 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:48:36.553174 28266 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 22:48:36.563251 28266 rpc_server.cc:307] RPC server started. Bound to: 127.10.198.2:43319
I20260925 22:48:36.563277 28394 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.198.2:43319 every 8 connection(s)
I20260925 22:48:36.564532 28266 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790376259650642-11032-0/minicluster-data/ts-1/data/info.pb
I20260925 22:48:36.570830 11032 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrWXm8X/build/debug/bin/kudu as pid 28266
I20260925 22:48:36.570989 11032 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790376259650642-11032-0/minicluster-data/ts-1/wal/instance
I20260925 22:48:36.571341 11032 external_mini_cluster.cc:1428] Setting key e5acc6d9b0442353b37ce4cfda8a89e9
Sep 25 22:48:36 dist-test-slave-5vdr krb5kdc[28025](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376516, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.198.2@KRBTEST.COM for kudu/127.10.198.62@KRBTEST.COM
I20260925 22:48:36.581575 28109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:48:36.566740 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.198.2:36875 (local address 127.10.198.62:33175)
0925 22:48:36.566907 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0925 22:48:36.566911 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:48:36.568190 (+ 1279us) server_negotiation.cc:407] Connection header received
0925 22:48:36.569184 (+ 994us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:48:36.569187 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:48:36.569251 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:48:36.569335 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:48:36.570722 (+ 1387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:48:36.571612 (+ 890us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:48:36.573374 (+ 1762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:48:36.573834 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:48:36.573984 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:48:36.577408 (+ 3424us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:48:36.577412 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:48:36.577414 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:48:36.577444 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:48:36.578979 (+ 1535us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:48:36.579700 (+ 721us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:48:36.579704 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:48:36.579706 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:48:36.579800 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:48:36.580284 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:48:36.580288 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:48:36.580290 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:48:36.580510 (+ 220us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:48:36.580641 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0925 22:48:36.581243 (+ 602us) server_negotiation.cc:299] Negotiation successful
0925 22:48:36.581387 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
WARNING: no policy specified for kudu/127.10.198.3@KRBTEST.COM; defaulting to no policy
I20260925 22:48:36.582535 28397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:48:36.567130 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.198.62:33175 (local address 127.10.198.2:36875)
0925 22:48:36.568010 (+ 880us) negotiation.cc:107] Waiting for socket to connect
0925 22:48:36.568069 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0925 22:48:36.568943 (+ 874us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:48:36.569527 (+ 584us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:48:36.569534 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:48:36.569964 (+ 430us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:48:36.570514 (+ 550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:48:36.570537 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:48:36.571797 (+ 1260us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:48:36.571802 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:48:36.573218 (+ 1416us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:48:36.573232 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:48:36.574021 (+ 789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:48:36.574024 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:48:36.574200 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:48:36.574899 (+ 699us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:48:36.574918 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:48:36.577155 (+ 2237us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:48:36.579139 (+ 1984us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:48:36.579146 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:48:36.579192 (+ 46us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:48:36.579534 (+ 342us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:48:36.579957 (+ 423us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:48:36.579961 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:48:36.579963 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:48:36.580161 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:48:36.580640 (+ 479us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:48:36.580650 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:48:36.580955 (+ 305us) client_negotiation.cc:769] Sending connection context
0925 22:48:36.581175 (+ 220us) client_negotiation.cc:240] Negotiation successful
0925 22:48:36.581562 (+ 387us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":619,"thread_start_us":130,"threads_started":1}
I20260925 22:48:36.584084 28395 heartbeater.cc:347] Connected to a master server at 127.10.198.62:33175
I20260925 22:48:36.584376 28395 heartbeater.cc:464] Registering TS with master...
I20260925 22:48:36.584925 28395 heartbeater.cc:511] Master 127.10.198.62:33175 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.198.3@KRBTEST.COM" created.
I20260925 22:48:36.586305 28070 ts_manager.cc:194] Registered new tserver with Master: 821cc55a7f9941c7949757511c90a2a7 (127.10.198.2:43319)
I20260925 22:48:36.587023 28070 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.198.2@KRBTEST.COM'} at 127.10.198.2:36875
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.198.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790376259650642-11032-0/krb5kdc/kudu_127.10.198.3.keytab.
Entry for principal kudu/127.10.198.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790376259650642-11032-0/krb5kdc/kudu_127.10.198.3.keytab.
I20260925 22:48:36.604409 11032 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrWXm8X/build/debug/bin/kudu
/tmp/dist-test-taskrWXm8X/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790376259650642-11032-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790376259650642-11032-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790376259650642-11032-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790376259650642-11032-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.198.3:0
--local_ip_for_outbound_sockets=127.10.198.3
--webserver_interface=127.10.198.3
--webserver_port=0
--tserver_master_addrs=127.10.198.62:33175
--keytab_file=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790376259650642-11032-0/krb5kdc/kudu_127.10.198.3.keytab
--principal=kudu/127.10.198.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.198.20:37837
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790376259650642-11032-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790376259650642-11032-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790376259650642-11032-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:48:36.722878 28400 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:48:36.723194 28400 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:48:36.723239 28400 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:48:36.727694 28400 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:48:36.727805 28400 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:48:36.727957 28400 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.198.3
I20260925 22:48:36.735033 28400 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.198.20:37837
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790376259650642-11032-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790376259650642-11032-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790376259650642-11032-0/krb5kdc/kudu_127.10.198.3.keytab
--principal=kudu/127.10.198.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.198.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790376259650642-11032-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.198.3
--webserver_port=0
--tserver_master_addrs=127.10.198.62:33175
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28400
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.198.3
--log_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790376259650642-11032-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:09:16 UTC on ccycloud.quasar-afmhyw.root.comops.site
build id 4673379
I20260925 22:48:36.736275 28400 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:48:36.737311 28400 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:48:36.747529 28405 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:48:36.747529 28406 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:48:36.747543 28408 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:48:36.747902 28400 server_base.cc:1034] running on GCE node
I20260925 22:48:36.748389 28400 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:48:36.749109 28400 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:48:36.750289 28400 hybrid_clock.cc:648] HybridClock initialized: now 1790376516750265 us; error 33 us; skew 500 ppm
Sep 25 22:48:36 dist-test-slave-5vdr krb5kdc[28025](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376516, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.198.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:48:36.753428 28400 init.cc:377] Logged in from keytab as kudu/127.10.198.3@KRBTEST.COM (short username kudu)
I20260925 22:48:36.754478 28400 webserver.cc:460] Webserver started at http://127.10.198.3:35771/ using document root <none> and password file <none>
I20260925 22:48:36.755172 28400 fs_manager.cc:359] Metadata directory not provided
I20260925 22:48:36.755261 28400 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:48:36.755461 28400 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:48:36.758059 28400 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790376259650642-11032-0/minicluster-data/ts-2/data/instance:
uuid: "851888664ec34d15a6a2ba2235d47db9"
format_stamp: "Formatted at 2026-09-25 22:48:36 on dist-test-slave-5vdr"
server_key: "77508d499431a0c96d51df8724478053"
server_key_iv: "091a5cf3f7374550cdbb2b56a0d01974"
server_key_version: "encryptionkey@0"
I20260925 22:48:36.758665 28400 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790376259650642-11032-0/minicluster-data/ts-2/wal/instance:
uuid: "851888664ec34d15a6a2ba2235d47db9"
format_stamp: "Formatted at 2026-09-25 22:48:36 on dist-test-slave-5vdr"
server_key: "77508d499431a0c96d51df8724478053"
server_key_iv: "091a5cf3f7374550cdbb2b56a0d01974"
server_key_version: "encryptionkey@0"
I20260925 22:48:36.762990 28400 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.005s
I20260925 22:48:36.765928 28415 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:48:36.767402 28400 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260925 22:48:36.767572 28400 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790376259650642-11032-0/minicluster-data/ts-2/data,/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790376259650642-11032-0/minicluster-data/ts-2/wal
uuid: "851888664ec34d15a6a2ba2235d47db9"
format_stamp: "Formatted at 2026-09-25 22:48:36 on dist-test-slave-5vdr"
server_key: "77508d499431a0c96d51df8724478053"
server_key_iv: "091a5cf3f7374550cdbb2b56a0d01974"
server_key_version: "encryptionkey@0"
I20260925 22:48:36.767709 28400 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790376259650642-11032-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790376259650642-11032-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790376259650642-11032-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:48:36.782102 28400 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:48:36.785727 28400 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:48:36.785976 28400 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:48:36.786645 28400 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 22:48:36.787547 28400 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:48:36.787598 28400 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:48:36.787664 28400 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:48:36.787705 28400 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 22:48:36.797461 28400 rpc_server.cc:307] RPC server started. Bound to: 127.10.198.3:33385
I20260925 22:48:36.797499 28528 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.198.3:33385 every 8 connection(s)
I20260925 22:48:36.798481 28400 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790376259650642-11032-0/minicluster-data/ts-2/data/info.pb
I20260925 22:48:36.801656 11032 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrWXm8X/build/debug/bin/kudu as pid 28400
I20260925 22:48:36.801823 11032 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790376259650642-11032-0/minicluster-data/ts-2/wal/instance
I20260925 22:48:36.802147 11032 external_mini_cluster.cc:1428] Setting key 5d7aa763be1b8ae3477bf5ad0e6daa79
Sep 25 22:48:36 dist-test-slave-5vdr krb5kdc[28025](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376516, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.198.3@KRBTEST.COM for kudu/127.10.198.62@KRBTEST.COM
I20260925 22:48:36.813675 28109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:48:36.800628 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.198.3:55427 (local address 127.10.198.62:33175)
0925 22:48:36.800797 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0925 22:48:36.800802 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:48:36.801862 (+ 1060us) server_negotiation.cc:407] Connection header received
0925 22:48:36.803378 (+ 1516us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:48:36.803382 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:48:36.803440 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:48:36.803518 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:48:36.804895 (+ 1377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:48:36.805473 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:48:36.806703 (+ 1230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:48:36.806924 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:48:36.807010 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:48:36.809889 (+ 2879us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:48:36.809893 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:48:36.809895 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:48:36.809924 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:48:36.811466 (+ 1542us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:48:36.812120 (+ 654us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:48:36.812123 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:48:36.812124 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:48:36.812210 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:48:36.812538 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:48:36.812540 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:48:36.812541 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:48:36.812731 (+ 190us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:48:36.812810 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0925 22:48:36.813423 (+ 613us) server_negotiation.cc:299] Negotiation successful
0925 22:48:36.813543 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260925 22:48:36.814607 28531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:48:36.801062 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.198.62:33175 (local address 127.10.198.3:55427)
0925 22:48:36.801615 (+ 553us) negotiation.cc:107] Waiting for socket to connect
0925 22:48:36.801666 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0925 22:48:36.803160 (+ 1494us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:48:36.803732 (+ 572us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:48:36.803740 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:48:36.804203 (+ 463us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:48:36.804712 (+ 509us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:48:36.804726 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:48:36.805613 (+ 887us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:48:36.805616 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:48:36.806578 (+ 962us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:48:36.806588 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:48:36.807065 (+ 477us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:48:36.807068 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:48:36.807250 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:48:36.807952 (+ 702us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:48:36.807970 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:48:36.809667 (+ 1697us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:48:36.811615 (+ 1948us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:48:36.811621 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:48:36.811636 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:48:36.812025 (+ 389us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:48:36.812295 (+ 270us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:48:36.812297 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:48:36.812299 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:48:36.812450 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:48:36.812830 (+ 380us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:48:36.812836 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:48:36.813122 (+ 286us) client_negotiation.cc:769] Sending connection context
0925 22:48:36.813318 (+ 196us) client_negotiation.cc:240] Negotiation successful
0925 22:48:36.813627 (+ 309us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":326,"thread_start_us":134,"threads_started":1}
I20260925 22:48:36.815873 28529 heartbeater.cc:347] Connected to a master server at 127.10.198.62:33175
I20260925 22:48:36.816154 28529 heartbeater.cc:464] Registering TS with master...
I20260925 22:48:36.816706 28529 heartbeater.cc:511] Master 127.10.198.62:33175 requested a full tablet report, sending...
I20260925 22:48:36.817785 28070 ts_manager.cc:194] Registered new tserver with Master: 851888664ec34d15a6a2ba2235d47db9 (127.10.198.3:33385)
I20260925 22:48:36.818431 28070 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.198.3@KRBTEST.COM'} at 127.10.198.3:55427
I20260925 22:48:36.828490 11032 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 22:48:36.838764 28109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:48:36.830669 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54178 (local address 127.10.198.62:33175)
0925 22:48:36.830935 (+ 266us) server_negotiation.cc:206] Beginning negotiation
0925 22:48:36.830939 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:48:36.831065 (+ 126us) server_negotiation.cc:407] Connection header received
0925 22:48:36.831255 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:48:36.831259 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:48:36.831312 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:48:36.831403 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:48:36.832239 (+ 836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:48:36.832791 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:48:36.833767 (+ 976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:48:36.833994 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:48:36.834108 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:48:36.835583 (+ 1475us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:48:36.835592 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:48:36.835595 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:48:36.835624 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:48:36.836979 (+ 1355us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:48:36.837435 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:48:36.837439 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:48:36.837441 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:48:36.837528 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:48:36.837805 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:48:36.837809 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:48:36.837811 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:48:36.838056 (+ 245us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:48:36.838203 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0925 22:48:36.838450 (+ 247us) server_negotiation.cc:299] Negotiation successful
0925 22:48:36.838579 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":163}
I20260925 22:48:36.842161 28070 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54178:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 22:48:36.845533 28070 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 22:48:36.862205 28540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:48:36.855408 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.198.3:33385 (local address 127.0.0.1:59774)
0925 22:48:36.856340 (+ 932us) negotiation.cc:107] Waiting for socket to connect
0925 22:48:36.856400 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0925 22:48:36.856576 (+ 176us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:48:36.857702 (+ 1126us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:48:36.857703 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:48:36.857718 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:48:36.857787 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:48:36.857791 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:48:36.858884 (+ 1093us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:48:36.858890 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:48:36.860541 (+ 1651us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:48:36.860552 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:48:36.861538 (+ 986us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:48:36.861542 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:48:36.861687 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:48:36.861732 (+ 45us) client_negotiation.cc:769] Sending connection context
0925 22:48:36.861890 (+ 158us) client_negotiation.cc:240] Negotiation successful
0925 22:48:36.862053 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":187,"mutex_wait_us":98,"thread_start_us":96,"threads_started":1}
I20260925 22:48:36.862218 28541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:48:36.855555 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59774 (local address 127.10.198.3:33385)
0925 22:48:36.856843 (+ 1288us) server_negotiation.cc:206] Beginning negotiation
0925 22:48:36.856857 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0925 22:48:36.856884 (+ 27us) server_negotiation.cc:407] Connection header received
0925 22:48:36.856969 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:48:36.856976 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:48:36.857185 (+ 209us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:48:36.857400 (+ 215us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:48:36.857892 (+ 492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:48:36.858735 (+ 843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:48:36.860702 (+ 1967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:48:36.861388 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:48:36.861531 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:48:36.861720 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0925 22:48:36.861923 (+ 203us) server_negotiation.cc:299] Negotiation successful
0925 22:48:36.862037 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":850,"thread_start_us":178,"threads_started":1}
I20260925 22:48:36.862498 28542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:48:36.856186 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.198.2:43319 (local address 127.0.0.1:51728)
0925 22:48:36.857338 (+ 1152us) negotiation.cc:107] Waiting for socket to connect
0925 22:48:36.857353 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 22:48:36.857443 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:48:36.857953 (+ 510us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:48:36.857956 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:48:36.857972 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:48:36.858044 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:48:36.858049 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:48:36.859410 (+ 1361us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:48:36.859413 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:48:36.860459 (+ 1046us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:48:36.860471 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:48:36.861895 (+ 1424us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:48:36.861899 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:48:36.862097 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:48:36.862183 (+ 86us) client_negotiation.cc:769] Sending connection context
0925 22:48:36.862259 (+ 76us) client_negotiation.cc:240] Negotiation successful
0925 22:48:36.862349 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1084,"mutex_wait_us":76,"thread_start_us":103,"threads_started":1}
I20260925 22:48:36.862728 28543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:48:36.856078 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41986 (local address 127.10.198.1:43261)
0925 22:48:36.856869 (+ 791us) server_negotiation.cc:206] Beginning negotiation
0925 22:48:36.856879 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 22:48:36.856904 (+ 25us) server_negotiation.cc:407] Connection header received
0925 22:48:36.856984 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:48:36.856989 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:48:36.857199 (+ 210us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:48:36.857370 (+ 171us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:48:36.857697 (+ 327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:48:36.858883 (+ 1186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:48:36.860822 (+ 1939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:48:36.861634 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:48:36.861735 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:48:36.861863 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0925 22:48:36.862440 (+ 577us) server_negotiation.cc:299] Negotiation successful
0925 22:48:36.862563 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":680,"thread_start_us":90,"threads_started":1}
I20260925 22:48:36.863292 28545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:48:36.856763 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51728 (local address 127.10.198.2:43319)
0925 22:48:36.857187 (+ 424us) server_negotiation.cc:206] Beginning negotiation
0925 22:48:36.857197 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 22:48:36.857542 (+ 345us) server_negotiation.cc:407] Connection header received
0925 22:48:36.857629 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:48:36.857634 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:48:36.857830 (+ 196us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:48:36.857985 (+ 155us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:48:36.858144 (+ 159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:48:36.859244 (+ 1100us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:48:36.860600 (+ 1356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:48:36.861498 (+ 898us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:48:36.861638 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:48:36.861818 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0925 22:48:36.863043 (+ 1225us) server_negotiation.cc:299] Negotiation successful
0925 22:48:36.863172 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":325,"thread_start_us":125,"threads_started":1}
I20260925 22:48:36.863025 28544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:48:36.855898 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.198.1:43261 (local address 127.0.0.1:41986)
0925 22:48:36.856477 (+ 579us) negotiation.cc:107] Waiting for socket to connect
0925 22:48:36.856491 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 22:48:36.856590 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:48:36.857358 (+ 768us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:48:36.857364 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:48:36.857389 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:48:36.857591 (+ 202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:48:36.857605 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:48:36.858999 (+ 1394us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:48:36.859002 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:48:36.860686 (+ 1684us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:48:36.860694 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:48:36.861913 (+ 1219us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:48:36.861916 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:48:36.862100 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:48:36.862188 (+ 88us) client_negotiation.cc:769] Sending connection context
0925 22:48:36.862259 (+ 71us) client_negotiation.cc:240] Negotiation successful
0925 22:48:36.862349 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":517,"mutex_wait_us":141,"spinlock_wait_cycles":12544,"thread_start_us":102,"threads_started":1}
I20260925 22:48:36.866209 28195 tablet_service.cc:1467] Processing CreateTablet for tablet 5a087457fb804eb7b3c0ce99e4a4a2b3 (DEFAULT_TABLE table=test-table [id=09f5d5de7ed44a3596b3a46b03eac929]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 22:48:36.866050 28463 tablet_service.cc:1467] Processing CreateTablet for tablet 5a087457fb804eb7b3c0ce99e4a4a2b3 (DEFAULT_TABLE table=test-table [id=09f5d5de7ed44a3596b3a46b03eac929]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 22:48:36.866361 28329 tablet_service.cc:1467] Processing CreateTablet for tablet 5a087457fb804eb7b3c0ce99e4a4a2b3 (DEFAULT_TABLE table=test-table [id=09f5d5de7ed44a3596b3a46b03eac929]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 22:48:36.867312 28195 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5a087457fb804eb7b3c0ce99e4a4a2b3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:48:36.867317 28329 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5a087457fb804eb7b3c0ce99e4a4a2b3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:48:36.867326 28463 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5a087457fb804eb7b3c0ce99e4a4a2b3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:48:36.874630 28546 tablet_bootstrap.cc:492] T 5a087457fb804eb7b3c0ce99e4a4a2b3 P d812b13d48aa401e94c7572985e507f3: Bootstrap starting.
I20260925 22:48:36.875985 28548 tablet_bootstrap.cc:492] T 5a087457fb804eb7b3c0ce99e4a4a2b3 P 821cc55a7f9941c7949757511c90a2a7: Bootstrap starting.
I20260925 22:48:36.876165 28547 tablet_bootstrap.cc:492] T 5a087457fb804eb7b3c0ce99e4a4a2b3 P 851888664ec34d15a6a2ba2235d47db9: Bootstrap starting.
I20260925 22:48:36.877326 28546 tablet_bootstrap.cc:654] T 5a087457fb804eb7b3c0ce99e4a4a2b3 P d812b13d48aa401e94c7572985e507f3: Neither blocks nor log segments found. Creating new log.
I20260925 22:48:36.878137 28546 log.cc:824] T 5a087457fb804eb7b3c0ce99e4a4a2b3 P d812b13d48aa401e94c7572985e507f3: Log is configured to *not* fsync() on all Append() calls
I20260925 22:48:36.878372 28548 tablet_bootstrap.cc:654] T 5a087457fb804eb7b3c0ce99e4a4a2b3 P 821cc55a7f9941c7949757511c90a2a7: Neither blocks nor log segments found. Creating new log.
I20260925 22:48:36.878372 28547 tablet_bootstrap.cc:654] T 5a087457fb804eb7b3c0ce99e4a4a2b3 P 851888664ec34d15a6a2ba2235d47db9: Neither blocks nor log segments found. Creating new log.
I20260925 22:48:36.879987 28547 log.cc:824] T 5a087457fb804eb7b3c0ce99e4a4a2b3 P 851888664ec34d15a6a2ba2235d47db9: Log is configured to *not* fsync() on all Append() calls
I20260925 22:48:36.879987 28548 log.cc:824] T 5a087457fb804eb7b3c0ce99e4a4a2b3 P 821cc55a7f9941c7949757511c90a2a7: Log is configured to *not* fsync() on all Append() calls
I20260925 22:48:36.881031 28546 tablet_bootstrap.cc:492] T 5a087457fb804eb7b3c0ce99e4a4a2b3 P d812b13d48aa401e94c7572985e507f3: No bootstrap required, opened a new log
I20260925 22:48:36.881301 28546 ts_tablet_manager.cc:1397] T 5a087457fb804eb7b3c0ce99e4a4a2b3 P d812b13d48aa401e94c7572985e507f3: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.004s
I20260925 22:48:36.882434 28547 tablet_bootstrap.cc:492] T 5a087457fb804eb7b3c0ce99e4a4a2b3 P 851888664ec34d15a6a2ba2235d47db9: No bootstrap required, opened a new log
I20260925 22:48:36.882434 28548 tablet_bootstrap.cc:492] T 5a087457fb804eb7b3c0ce99e4a4a2b3 P 821cc55a7f9941c7949757511c90a2a7: No bootstrap required, opened a new log
I20260925 22:48:36.882661 28547 ts_tablet_manager.cc:1397] T 5a087457fb804eb7b3c0ce99e4a4a2b3 P 851888664ec34d15a6a2ba2235d47db9: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260925 22:48:36.882661 28548 ts_tablet_manager.cc:1397] T 5a087457fb804eb7b3c0ce99e4a4a2b3 P 821cc55a7f9941c7949757511c90a2a7: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260925 22:48:36.885970 28546 raft_consensus.cc:348] T 5a087457fb804eb7b3c0ce99e4a4a2b3 P d812b13d48aa401e94c7572985e507f3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "851888664ec34d15a6a2ba2235d47db9" member_type: VOTER last_known_addr { host: "127.10.198.3" port: 33385 } } peers { permanent_uuid: "821cc55a7f9941c7949757511c90a2a7" member_type: VOTER last_known_addr { host: "127.10.198.2" port: 43319 } } peers { permanent_uuid: "d812b13d48aa401e94c7572985e507f3" member_type: VOTER last_known_addr { host: "127.10.198.1" port: 43261 } }
I20260925 22:48:36.886344 28546 raft_consensus.cc:374] T 5a087457fb804eb7b3c0ce99e4a4a2b3 P d812b13d48aa401e94c7572985e507f3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:48:36.886415 28546 raft_consensus.cc:729] T 5a087457fb804eb7b3c0ce99e4a4a2b3 P d812b13d48aa401e94c7572985e507f3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d812b13d48aa401e94c7572985e507f3, State: Initialized, Role: FOLLOWER
I20260925 22:48:36.886973 28546 consensus_queue.cc:260] T 5a087457fb804eb7b3c0ce99e4a4a2b3 P d812b13d48aa401e94c7572985e507f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "851888664ec34d15a6a2ba2235d47db9" member_type: VOTER last_known_addr { host: "127.10.198.3" port: 33385 } } peers { permanent_uuid: "821cc55a7f9941c7949757511c90a2a7" member_type: VOTER last_known_addr { host: "127.10.198.2" port: 43319 } } peers { permanent_uuid: "d812b13d48aa401e94c7572985e507f3" member_type: VOTER last_known_addr { host: "127.10.198.1" port: 43261 } }
I20260925 22:48:36.886895 28548 raft_consensus.cc:348] T 5a087457fb804eb7b3c0ce99e4a4a2b3 P 821cc55a7f9941c7949757511c90a2a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "851888664ec34d15a6a2ba2235d47db9" member_type: VOTER last_known_addr { host: "127.10.198.3" port: 33385 } } peers { permanent_uuid: "821cc55a7f9941c7949757511c90a2a7" member_type: VOTER last_known_addr { host: "127.10.198.2" port: 43319 } } peers { permanent_uuid: "d812b13d48aa401e94c7572985e507f3" member_type: VOTER last_known_addr { host: "127.10.198.1" port: 43261 } }
I20260925 22:48:36.886895 28547 raft_consensus.cc:348] T 5a087457fb804eb7b3c0ce99e4a4a2b3 P 851888664ec34d15a6a2ba2235d47db9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "851888664ec34d15a6a2ba2235d47db9" member_type: VOTER last_known_addr { host: "127.10.198.3" port: 33385 } } peers { permanent_uuid: "821cc55a7f9941c7949757511c90a2a7" member_type: VOTER last_known_addr { host: "127.10.198.2" port: 43319 } } peers { permanent_uuid: "d812b13d48aa401e94c7572985e507f3" member_type: VOTER last_known_addr { host: "127.10.198.1" port: 43261 } }
I20260925 22:48:36.887212 28547 raft_consensus.cc:374] T 5a087457fb804eb7b3c0ce99e4a4a2b3 P 851888664ec34d15a6a2ba2235d47db9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:48:36.887212 28548 raft_consensus.cc:374] T 5a087457fb804eb7b3c0ce99e4a4a2b3 P 821cc55a7f9941c7949757511c90a2a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:48:36.887269 28548 raft_consensus.cc:729] T 5a087457fb804eb7b3c0ce99e4a4a2b3 P 821cc55a7f9941c7949757511c90a2a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 821cc55a7f9941c7949757511c90a2a7, State: Initialized, Role: FOLLOWER
I20260925 22:48:36.887291 28547 raft_consensus.cc:729] T 5a087457fb804eb7b3c0ce99e4a4a2b3 P 851888664ec34d15a6a2ba2235d47db9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 851888664ec34d15a6a2ba2235d47db9, State: Initialized, Role: FOLLOWER
I20260925 22:48:36.887763 28547 consensus_queue.cc:260] T 5a087457fb804eb7b3c0ce99e4a4a2b3 P 851888664ec34d15a6a2ba2235d47db9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "851888664ec34d15a6a2ba2235d47db9" member_type: VOTER last_known_addr { host: "127.10.198.3" port: 33385 } } peers { permanent_uuid: "821cc55a7f9941c7949757511c90a2a7" member_type: VOTER last_known_addr { host: "127.10.198.2" port: 43319 } } peers { permanent_uuid: "d812b13d48aa401e94c7572985e507f3" member_type: VOTER last_known_addr { host: "127.10.198.1" port: 43261 } }
I20260925 22:48:36.887763 28548 consensus_queue.cc:260] T 5a087457fb804eb7b3c0ce99e4a4a2b3 P 821cc55a7f9941c7949757511c90a2a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "851888664ec34d15a6a2ba2235d47db9" member_type: VOTER last_known_addr { host: "127.10.198.3" port: 33385 } } peers { permanent_uuid: "821cc55a7f9941c7949757511c90a2a7" member_type: VOTER last_known_addr { host: "127.10.198.2" port: 43319 } } peers { permanent_uuid: "d812b13d48aa401e94c7572985e507f3" member_type: VOTER last_known_addr { host: "127.10.198.1" port: 43261 } }
I20260925 22:48:36.888319 28546 ts_tablet_manager.cc:1428] T 5a087457fb804eb7b3c0ce99e4a4a2b3 P d812b13d48aa401e94c7572985e507f3: Time spent starting tablet: real 0.007s user 0.008s sys 0.000s
I20260925 22:48:36.888655 28529 heartbeater.cc:503] Master 127.10.198.62:33175 was elected leader, sending a full tablet report...
I20260925 22:48:36.888829 28547 ts_tablet_manager.cc:1428] T 5a087457fb804eb7b3c0ce99e4a4a2b3 P 851888664ec34d15a6a2ba2235d47db9: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260925 22:48:36.888970 28395 heartbeater.cc:503] Master 127.10.198.62:33175 was elected leader, sending a full tablet report...
I20260925 22:48:36.888829 28548 ts_tablet_manager.cc:1428] T 5a087457fb804eb7b3c0ce99e4a4a2b3 P 821cc55a7f9941c7949757511c90a2a7: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260925 22:48:36.889946 28261 heartbeater.cc:503] Master 127.10.198.62:33175 was elected leader, sending a full tablet report...
I20260925 22:48:36.941375 28554 raft_consensus.cc:482] T 5a087457fb804eb7b3c0ce99e4a4a2b3 P 821cc55a7f9941c7949757511c90a2a7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 22:48:36.941659 28554 raft_consensus.cc:504] T 5a087457fb804eb7b3c0ce99e4a4a2b3 P 821cc55a7f9941c7949757511c90a2a7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "851888664ec34d15a6a2ba2235d47db9" member_type: VOTER last_known_addr { host: "127.10.198.3" port: 33385 } } peers { permanent_uuid: "821cc55a7f9941c7949757511c90a2a7" member_type: VOTER last_known_addr { host: "127.10.198.2" port: 43319 } } peers { permanent_uuid: "d812b13d48aa401e94c7572985e507f3" member_type: VOTER last_known_addr { host: "127.10.198.1" port: 43261 } }
I20260925 22:48:36.943141 28554 leader_election.cc:288] T 5a087457fb804eb7b3c0ce99e4a4a2b3 P 821cc55a7f9941c7949757511c90a2a7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 851888664ec34d15a6a2ba2235d47db9 (127.10.198.3:33385), d812b13d48aa401e94c7572985e507f3 (127.10.198.1:43261)
I20260925 22:48:36.947783 28541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:48:36.943702 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.198.2:46103 (local address 127.10.198.3:33385)
0925 22:48:36.944035 (+ 333us) server_negotiation.cc:206] Beginning negotiation
0925 22:48:36.944042 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 22:48:36.944134 (+ 92us) server_negotiation.cc:407] Connection header received
0925 22:48:36.944252 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:48:36.944255 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:48:36.944327 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:48:36.944431 (+ 104us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:48:36.944754 (+ 323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:48:36.945355 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:48:36.946531 (+ 1176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:48:36.947078 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:48:36.947248 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:48:36.947309 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0925 22:48:36.947520 (+ 211us) server_negotiation.cc:299] Negotiation successful
0925 22:48:36.947639 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":249}
I20260925 22:48:36.947786 28397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:48:36.943507 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.198.1:43261 (local address 127.10.198.2:34817)
0925 22:48:36.943739 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0925 22:48:36.943756 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 22:48:36.943911 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:48:36.944377 (+ 466us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:48:36.944383 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:48:36.944409 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:48:36.944547 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:48:36.944555 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:48:36.945355 (+ 800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:48:36.945360 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:48:36.946406 (+ 1046us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:48:36.946418 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:48:36.947220 (+ 802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:48:36.947225 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:48:36.947358 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:48:36.947438 (+ 80us) client_negotiation.cc:769] Sending connection context
0925 22:48:36.947548 (+ 110us) client_negotiation.cc:240] Negotiation successful
0925 22:48:36.947638 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":136,"mutex_wait_us":94}
I20260925 22:48:36.947809 28543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:48:36.943898 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.198.2:34817 (local address 127.10.198.1:43261)
0925 22:48:36.944125 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0925 22:48:36.944130 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:48:36.944144 (+ 14us) server_negotiation.cc:407] Connection header received
0925 22:48:36.944203 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:48:36.944216 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:48:36.944272 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:48:36.944330 (+ 58us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:48:36.944691 (+ 361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:48:36.945219 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:48:36.946582 (+ 1363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:48:36.947052 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:48:36.947191 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:48:36.947265 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0925 22:48:36.947600 (+ 335us) server_negotiation.cc:299] Negotiation successful
0925 22:48:36.947700 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":118}
I20260925 22:48:36.947798 28555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:48:36.943507 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.198.3:33385 (local address 127.10.198.2:46103)
0925 22:48:36.944041 (+ 534us) negotiation.cc:107] Waiting for socket to connect
0925 22:48:36.944055 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 22:48:36.944149 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:48:36.944470 (+ 321us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:48:36.944472 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:48:36.944488 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:48:36.944583 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:48:36.944588 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:48:36.945486 (+ 898us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:48:36.945489 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:48:36.946374 (+ 885us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:48:36.946387 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:48:36.947249 (+ 862us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:48:36.947252 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:48:36.947361 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:48:36.947403 (+ 42us) client_negotiation.cc:769] Sending connection context
0925 22:48:36.947469 (+ 66us) client_negotiation.cc:240] Negotiation successful
0925 22:48:36.947578 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":467,"mutex_wait_us":44,"thread_start_us":151,"threads_started":1}
I20260925 22:48:36.948720 28215 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a087457fb804eb7b3c0ce99e4a4a2b3" candidate_uuid: "821cc55a7f9941c7949757511c90a2a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d812b13d48aa401e94c7572985e507f3" is_pre_election: true
I20260925 22:48:36.948966 28483 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a087457fb804eb7b3c0ce99e4a4a2b3" candidate_uuid: "821cc55a7f9941c7949757511c90a2a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "851888664ec34d15a6a2ba2235d47db9" is_pre_election: true
I20260925 22:48:36.949146 28215 raft_consensus.cc:2454] T 5a087457fb804eb7b3c0ce99e4a4a2b3 P d812b13d48aa401e94c7572985e507f3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 821cc55a7f9941c7949757511c90a2a7 in term 0.
I20260925 22:48:36.949331 28483 raft_consensus.cc:2454] T 5a087457fb804eb7b3c0ce99e4a4a2b3 P 851888664ec34d15a6a2ba2235d47db9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 821cc55a7f9941c7949757511c90a2a7 in term 0.
I20260925 22:48:36.949857 28282 leader_election.cc:302] T 5a087457fb804eb7b3c0ce99e4a4a2b3 P 821cc55a7f9941c7949757511c90a2a7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 821cc55a7f9941c7949757511c90a2a7, 851888664ec34d15a6a2ba2235d47db9, d812b13d48aa401e94c7572985e507f3; no voters:
I20260925 22:48:36.950196 28554 raft_consensus.cc:2781] T 5a087457fb804eb7b3c0ce99e4a4a2b3 P 821cc55a7f9941c7949757511c90a2a7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 22:48:36.950294 28554 raft_consensus.cc:482] T 5a087457fb804eb7b3c0ce99e4a4a2b3 P 821cc55a7f9941c7949757511c90a2a7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 22:48:36.950338 28554 raft_consensus.cc:3037] T 5a087457fb804eb7b3c0ce99e4a4a2b3 P 821cc55a7f9941c7949757511c90a2a7 [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:48:36.951452 28554 raft_consensus.cc:504] T 5a087457fb804eb7b3c0ce99e4a4a2b3 P 821cc55a7f9941c7949757511c90a2a7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "851888664ec34d15a6a2ba2235d47db9" member_type: VOTER last_known_addr { host: "127.10.198.3" port: 33385 } } peers { permanent_uuid: "821cc55a7f9941c7949757511c90a2a7" member_type: VOTER last_known_addr { host: "127.10.198.2" port: 43319 } } peers { permanent_uuid: "d812b13d48aa401e94c7572985e507f3" member_type: VOTER last_known_addr { host: "127.10.198.1" port: 43261 } }
I20260925 22:48:36.951932 28554 leader_election.cc:288] T 5a087457fb804eb7b3c0ce99e4a4a2b3 P 821cc55a7f9941c7949757511c90a2a7 [CANDIDATE]: Term 1 election: Requested vote from peers 851888664ec34d15a6a2ba2235d47db9 (127.10.198.3:33385), d812b13d48aa401e94c7572985e507f3 (127.10.198.1:43261)
I20260925 22:48:36.952405 28483 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a087457fb804eb7b3c0ce99e4a4a2b3" candidate_uuid: "821cc55a7f9941c7949757511c90a2a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "851888664ec34d15a6a2ba2235d47db9"
I20260925 22:48:36.952428 28215 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a087457fb804eb7b3c0ce99e4a4a2b3" candidate_uuid: "821cc55a7f9941c7949757511c90a2a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d812b13d48aa401e94c7572985e507f3"
I20260925 22:48:36.952534 28483 raft_consensus.cc:3037] T 5a087457fb804eb7b3c0ce99e4a4a2b3 P 851888664ec34d15a6a2ba2235d47db9 [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:48:36.952587 28215 raft_consensus.cc:3037] T 5a087457fb804eb7b3c0ce99e4a4a2b3 P d812b13d48aa401e94c7572985e507f3 [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:48:36.953751 28215 raft_consensus.cc:2454] T 5a087457fb804eb7b3c0ce99e4a4a2b3 P d812b13d48aa401e94c7572985e507f3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 821cc55a7f9941c7949757511c90a2a7 in term 1.
I20260925 22:48:36.954141 28483 raft_consensus.cc:2454] T 5a087457fb804eb7b3c0ce99e4a4a2b3 P 851888664ec34d15a6a2ba2235d47db9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 821cc55a7f9941c7949757511c90a2a7 in term 1.
I20260925 22:48:36.954216 28284 leader_election.cc:302] T 5a087457fb804eb7b3c0ce99e4a4a2b3 P 821cc55a7f9941c7949757511c90a2a7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 821cc55a7f9941c7949757511c90a2a7, d812b13d48aa401e94c7572985e507f3; no voters:
I20260925 22:48:36.954478 28554 raft_consensus.cc:2781] T 5a087457fb804eb7b3c0ce99e4a4a2b3 P 821cc55a7f9941c7949757511c90a2a7 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 22:48:36.954775 28554 raft_consensus.cc:686] T 5a087457fb804eb7b3c0ce99e4a4a2b3 P 821cc55a7f9941c7949757511c90a2a7 [term 1 LEADER]: Becoming Leader. State: Replica: 821cc55a7f9941c7949757511c90a2a7, State: Running, Role: LEADER
I20260925 22:48:36.955188 28554 consensus_queue.cc:237] T 5a087457fb804eb7b3c0ce99e4a4a2b3 P 821cc55a7f9941c7949757511c90a2a7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "851888664ec34d15a6a2ba2235d47db9" member_type: VOTER last_known_addr { host: "127.10.198.3" port: 33385 } } peers { permanent_uuid: "821cc55a7f9941c7949757511c90a2a7" member_type: VOTER last_known_addr { host: "127.10.198.2" port: 43319 } } peers { permanent_uuid: "d812b13d48aa401e94c7572985e507f3" member_type: VOTER last_known_addr { host: "127.10.198.1" port: 43261 } }
I20260925 22:48:36.959345 28069 catalog_manager.cc:5495] T 5a087457fb804eb7b3c0ce99e4a4a2b3 P 821cc55a7f9941c7949757511c90a2a7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 821cc55a7f9941c7949757511c90a2a7 (127.10.198.2). New cstate: current_term: 1 leader_uuid: "821cc55a7f9941c7949757511c90a2a7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "851888664ec34d15a6a2ba2235d47db9" member_type: VOTER last_known_addr { host: "127.10.198.3" port: 33385 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "821cc55a7f9941c7949757511c90a2a7" member_type: VOTER last_known_addr { host: "127.10.198.2" port: 43319 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d812b13d48aa401e94c7572985e507f3" member_type: VOTER last_known_addr { host: "127.10.198.1" port: 43261 } health_report { overall_health: UNKNOWN } } }
I20260925 22:48:36.993475 28545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:48:36.989409 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51730 (local address 127.10.198.2:43319)
0925 22:48:36.989559 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0925 22:48:36.989563 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:48:36.989579 (+ 16us) server_negotiation.cc:407] Connection header received
0925 22:48:36.989762 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:48:36.989765 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:48:36.989826 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:48:36.989949 (+ 123us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 22:48:36.990208 (+ 259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:48:36.990736 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:48:36.991701 (+ 965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:48:36.991982 (+ 281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:48:36.992120 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:48:36.992507 (+ 387us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 22:48:36.992981 (+ 474us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 22:48:36.993084 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0925 22:48:36.993191 (+ 107us) server_negotiation.cc:299] Negotiation successful
0925 22:48:36.993253 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260925 22:48:37.000504 28483 raft_consensus.cc:1261] T 5a087457fb804eb7b3c0ce99e4a4a2b3 P 851888664ec34d15a6a2ba2235d47db9 [term 1 FOLLOWER]: Refusing update from remote peer 821cc55a7f9941c7949757511c90a2a7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 22:48:37.000557 28215 raft_consensus.cc:1261] T 5a087457fb804eb7b3c0ce99e4a4a2b3 P d812b13d48aa401e94c7572985e507f3 [term 1 FOLLOWER]: Refusing update from remote peer 821cc55a7f9941c7949757511c90a2a7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 22:48:37.001475 28554 consensus_queue.cc:1035] T 5a087457fb804eb7b3c0ce99e4a4a2b3 P 821cc55a7f9941c7949757511c90a2a7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "851888664ec34d15a6a2ba2235d47db9" member_type: VOTER last_known_addr { host: "127.10.198.3" port: 33385 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 22:48:37.001735 28556 consensus_queue.cc:1035] T 5a087457fb804eb7b3c0ce99e4a4a2b3 P 821cc55a7f9941c7949757511c90a2a7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d812b13d48aa401e94c7572985e507f3" member_type: VOTER last_known_addr { host: "127.10.198.1" port: 43261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 22:48:37.010319 28559 mvcc.cc:205] Tried to move back new op lower bound from 7333382213625032704 to 7333382213452210176. Current Snapshot: MvccSnapshot[applied={T|T < 7333382213625032704}]
I20260925 22:48:37.011097 28560 mvcc.cc:205] Tried to move back new op lower bound from 7333382213625032704 to 7333382213452210176. Current Snapshot: MvccSnapshot[applied={T|T < 7333382213625032704}]
I20260925 22:48:37.023057 28070 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54178:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 22:48:37.023250 28070 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54178:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 22:48:37.026225 28070 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e124597a34ed4eaea13ca5444fc3cea8: Sending DeleteTablet for 3 replicas of tablet 5a087457fb804eb7b3c0ce99e4a4a2b3
I20260925 22:48:37.027230 28195 tablet_service.cc:1514] Processing DeleteTablet for tablet 5a087457fb804eb7b3c0ce99e4a4a2b3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 22:48:37 UTC) from {username='kudu', principal='kudu/127.10.198.62@KRBTEST.COM'} at 127.0.0.1:41986
I20260925 22:48:37.027271 28463 tablet_service.cc:1514] Processing DeleteTablet for tablet 5a087457fb804eb7b3c0ce99e4a4a2b3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 22:48:37 UTC) from {username='kudu', principal='kudu/127.10.198.62@KRBTEST.COM'} at 127.0.0.1:59774
I20260925 22:48:37.027269 28329 tablet_service.cc:1514] Processing DeleteTablet for tablet 5a087457fb804eb7b3c0ce99e4a4a2b3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 22:48:37 UTC) from {username='kudu', principal='kudu/127.10.198.62@KRBTEST.COM'} at 127.0.0.1:51728
I20260925 22:48:37.027915 28570 tablet_replica.cc:330] T 5a087457fb804eb7b3c0ce99e4a4a2b3 P 851888664ec34d15a6a2ba2235d47db9: stopping tablet replica
I20260925 22:48:37.027961 28569 tablet_replica.cc:330] T 5a087457fb804eb7b3c0ce99e4a4a2b3 P 821cc55a7f9941c7949757511c90a2a7: stopping tablet replica
I20260925 22:48:37.028287 28570 raft_consensus.cc:2229] T 5a087457fb804eb7b3c0ce99e4a4a2b3 P 851888664ec34d15a6a2ba2235d47db9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 22:48:37.028417 28569 raft_consensus.cc:2229] T 5a087457fb804eb7b3c0ce99e4a4a2b3 P 821cc55a7f9941c7949757511c90a2a7 [term 1 LEADER]: Raft consensus shutting down.
I20260925 22:48:37.028674 28570 raft_consensus.cc:2258] T 5a087457fb804eb7b3c0ce99e4a4a2b3 P 851888664ec34d15a6a2ba2235d47db9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 22:48:37.028980 11032 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrWXm8X/build/debug/bin/kudu with pid 28132
I20260925 22:48:37.029121 28571 tablet_replica.cc:330] T 5a087457fb804eb7b3c0ce99e4a4a2b3 P d812b13d48aa401e94c7572985e507f3: stopping tablet replica
I20260925 22:48:37.029409 28571 raft_consensus.cc:2229] T 5a087457fb804eb7b3c0ce99e4a4a2b3 P d812b13d48aa401e94c7572985e507f3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 22:48:37.031777 28570 ts_tablet_manager.cc:1905] T 5a087457fb804eb7b3c0ce99e4a4a2b3 P 851888664ec34d15a6a2ba2235d47db9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 22:48:37.034974 28570 ts_tablet_manager.cc:1918] T 5a087457fb804eb7b3c0ce99e4a4a2b3 P 851888664ec34d15a6a2ba2235d47db9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 22:48:37.035090 28570 log.cc:1196] T 5a087457fb804eb7b3c0ce99e4a4a2b3 P 851888664ec34d15a6a2ba2235d47db9: Deleting WAL directory at /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790376259650642-11032-0/minicluster-data/ts-2/wal/wals/5a087457fb804eb7b3c0ce99e4a4a2b3
I20260925 22:48:37.035454 28570 ts_tablet_manager.cc:1939] T 5a087457fb804eb7b3c0ce99e4a4a2b3 P 851888664ec34d15a6a2ba2235d47db9: Deleting consensus metadata
I20260925 22:48:37.037096 28054 catalog_manager.cc:4841] TS 851888664ec34d15a6a2ba2235d47db9 (127.10.198.3:33385): tablet 5a087457fb804eb7b3c0ce99e4a4a2b3 (table test-table [id=09f5d5de7ed44a3596b3a46b03eac929]) successfully deleted
I20260925 22:48:37.037760 28569 raft_consensus.cc:2258] T 5a087457fb804eb7b3c0ce99e4a4a2b3 P 821cc55a7f9941c7949757511c90a2a7 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260925 22:48:37.038970 28056 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.10.198.1:43261 (error 108)
I20260925 22:48:37.039687 28569 ts_tablet_manager.cc:1905] T 5a087457fb804eb7b3c0ce99e4a4a2b3 P 821cc55a7f9941c7949757511c90a2a7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 22:48:37.039875 11032 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrWXm8X/build/debug/bin/kudu with pid 28266
I20260925 22:48:37.042043 28544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:48:37.041631 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.198.1:43261 (local address 127.0.0.1:41996)
0925 22:48:37.041820 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0925 22:48:37.041921 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.198.1:43261: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
W20260925 22:48:37.042241 28056 catalog_manager.cc:4584] TS d812b13d48aa401e94c7572985e507f3 (127.10.198.1:43261): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 5a087457fb804eb7b3c0ce99e4a4a2b3: Network error: Client connection negotiation failed: client connection to 127.10.198.1:43261: connect: Connection refused (error 111)
I20260925 22:48:37.046550 28544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:48:37.046218 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.198.2:43319 (local address 127.0.0.1:51732)
0925 22:48:37.046368 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0925 22:48:37.046438 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.198.2:43319: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260925 22:48:37.046691 11032 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrWXm8X/build/debug/bin/kudu with pid 28400
I20260925 22:48:37.052726 11032 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrWXm8X/build/debug/bin/kudu with pid 28038
2026-09-25T22:48:37Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (2419 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-09-25T22:48:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T22:48:37Z Disabled control of system clock
I20260925 22:48:37.085508 11032 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrWXm8X/build/debug/bin/kudu
/tmp/dist-test-taskrWXm8X/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790376259650642-11032-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790376259650642-11032-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790376259650642-11032-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790376259650642-11032-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.198.62:33447
--webserver_interface=127.10.198.62
--webserver_port=0
--builtin_ntp_servers=127.10.198.20:39847
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.198.62:33447
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260925 22:48:37.207502 28576 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:48:37.207832 28576 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:48:37.207943 28576 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:48:37.212347 28576 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 22:48:37.212430 28576 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:48:37.212448 28576 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:48:37.212467 28576 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 22:48:37.212484 28576 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 22:48:37.217865 28576 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.198.20:39847
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790376259650642-11032-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790376259650642-11032-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.198.62:33447
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.198.62:33447
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790376259650642-11032-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.198.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28576
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790376259650642-11032-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:09:16 UTC on ccycloud.quasar-afmhyw.root.comops.site
build id 4673379
I20260925 22:48:37.219062 28576 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:48:37.220628 28576 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:48:37.230634 28581 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:48:37.230634 28582 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:48:37.230646 28584 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:48:37.230967 28576 server_base.cc:1034] running on GCE node
I20260925 22:48:37.232120 28576 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:48:37.233534 28576 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:48:37.234777 28576 hybrid_clock.cc:648] HybridClock initialized: now 1790376517234761 us; error 43 us; skew 500 ppm
I20260925 22:48:37.237037 28576 webserver.cc:460] Webserver started at http://127.10.198.62:46695/ using document root <none> and password file <none>
I20260925 22:48:37.237709 28576 fs_manager.cc:359] Metadata directory not provided
I20260925 22:48:37.237816 28576 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:48:37.238062 28576 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:48:37.240474 28576 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790376259650642-11032-0/minicluster-data/master-0/data/instance:
uuid: "97282695dc384199a87add8d91dc30e9"
format_stamp: "Formatted at 2026-09-25 22:48:37 on dist-test-slave-5vdr"
server_key: "287dce46b48a236e447e718f64c18687"
server_key_iv: "2ae98f76f8f6e0c067978b17b5daf6d7"
server_key_version: "encryptionkey@0"
I20260925 22:48:37.241179 28576 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790376259650642-11032-0/minicluster-data/master-0/wal/instance:
uuid: "97282695dc384199a87add8d91dc30e9"
format_stamp: "Formatted at 2026-09-25 22:48:37 on dist-test-slave-5vdr"
server_key: "287dce46b48a236e447e718f64c18687"
server_key_iv: "2ae98f76f8f6e0c067978b17b5daf6d7"
server_key_version: "encryptionkey@0"
I20260925 22:48:37.245610 28576 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260925 22:48:37.248575 28590 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:48:37.250083 28576 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 22:48:37.250267 28576 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790376259650642-11032-0/minicluster-data/master-0/data,/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790376259650642-11032-0/minicluster-data/master-0/wal
uuid: "97282695dc384199a87add8d91dc30e9"
format_stamp: "Formatted at 2026-09-25 22:48:37 on dist-test-slave-5vdr"
server_key: "287dce46b48a236e447e718f64c18687"
server_key_iv: "2ae98f76f8f6e0c067978b17b5daf6d7"
server_key_version: "encryptionkey@0"
I20260925 22:48:37.250407 28576 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790376259650642-11032-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790376259650642-11032-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790376259650642-11032-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:48:37.265884 28576 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:48:37.266664 28576 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:48:37.266891 28576 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:48:37.274580 28576 rpc_server.cc:307] RPC server started. Bound to: 127.10.198.62:33447
I20260925 22:48:37.274627 28642 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.198.62:33447 every 8 connection(s)
I20260925 22:48:37.275668 28576 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790376259650642-11032-0/minicluster-data/master-0/data/info.pb
I20260925 22:48:37.279528 28643 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:48:37.282555 11032 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrWXm8X/build/debug/bin/kudu as pid 28576
I20260925 22:48:37.282724 11032 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790376259650642-11032-0/minicluster-data/master-0/wal/instance
I20260925 22:48:37.283067 11032 external_mini_cluster.cc:1428] Setting key 0257e46c9ea009446e545ba54eebacad
I20260925 22:48:37.287532 28643 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 97282695dc384199a87add8d91dc30e9: Bootstrap starting.
I20260925 22:48:37.290591 28643 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 97282695dc384199a87add8d91dc30e9: Neither blocks nor log segments found. Creating new log.
I20260925 22:48:37.291654 28643 log.cc:824] T 00000000000000000000000000000000 P 97282695dc384199a87add8d91dc30e9: Log is configured to *not* fsync() on all Append() calls
I20260925 22:48:37.293932 28646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:48:37.284798 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58076 (local address 127.10.198.62:33447)
0925 22:48:37.285485 (+ 687us) server_negotiation.cc:206] Beginning negotiation
0925 22:48:37.285502 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0925 22:48:37.285540 (+ 38us) server_negotiation.cc:407] Connection header received
0925 22:48:37.286354 (+ 814us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:48:37.286377 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:48:37.286823 (+ 446us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:48:37.287142 (+ 319us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:48:37.287577 (+ 435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:48:37.288703 (+ 1126us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:48:37.290120 (+ 1417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:48:37.290649 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:48:37.290867 (+ 218us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:48:37.291810 (+ 943us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:48:37.291818 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:48:37.291838 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 22:48:37.292410 (+ 572us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:48:37.292493 (+ 83us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 22:48:37.292525 (+ 32us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:48:37.292654 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0925 22:48:37.292840 (+ 186us) server_negotiation.cc:299] Negotiation successful
0925 22:48:37.293160 (+ 320us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":297,"thread_start_us":149,"threads_started":1}
I20260925 22:48:37.294135 28643 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 97282695dc384199a87add8d91dc30e9: No bootstrap required, opened a new log
I20260925 22:48:37.297933 28643 raft_consensus.cc:348] T 00000000000000000000000000000000 P 97282695dc384199a87add8d91dc30e9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "97282695dc384199a87add8d91dc30e9" member_type: VOTER last_known_addr { host: "127.10.198.62" port: 33447 } }
I20260925 22:48:37.298218 28643 raft_consensus.cc:374] T 00000000000000000000000000000000 P 97282695dc384199a87add8d91dc30e9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:48:37.298280 28643 raft_consensus.cc:729] T 00000000000000000000000000000000 P 97282695dc384199a87add8d91dc30e9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 97282695dc384199a87add8d91dc30e9, State: Initialized, Role: FOLLOWER
I20260925 22:48:37.298791 28643 consensus_queue.cc:260] T 00000000000000000000000000000000 P 97282695dc384199a87add8d91dc30e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "97282695dc384199a87add8d91dc30e9" member_type: VOTER last_known_addr { host: "127.10.198.62" port: 33447 } }
I20260925 22:48:37.298938 28643 raft_consensus.cc:388] T 00000000000000000000000000000000 P 97282695dc384199a87add8d91dc30e9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 22:48:37.299031 28643 raft_consensus.cc:482] T 00000000000000000000000000000000 P 97282695dc384199a87add8d91dc30e9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 22:48:37.299137 28643 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 97282695dc384199a87add8d91dc30e9 [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:48:37.300269 28643 raft_consensus.cc:504] T 00000000000000000000000000000000 P 97282695dc384199a87add8d91dc30e9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "97282695dc384199a87add8d91dc30e9" member_type: VOTER last_known_addr { host: "127.10.198.62" port: 33447 } }
I20260925 22:48:37.300671 28643 leader_election.cc:302] T 00000000000000000000000000000000 P 97282695dc384199a87add8d91dc30e9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 97282695dc384199a87add8d91dc30e9; no voters:
I20260925 22:48:37.301019 28643 leader_election.cc:288] T 00000000000000000000000000000000 P 97282695dc384199a87add8d91dc30e9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 22:48:37.301154 28648 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 97282695dc384199a87add8d91dc30e9 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 22:48:37.301429 28648 raft_consensus.cc:686] T 00000000000000000000000000000000 P 97282695dc384199a87add8d91dc30e9 [term 1 LEADER]: Becoming Leader. State: Replica: 97282695dc384199a87add8d91dc30e9, State: Running, Role: LEADER
I20260925 22:48:37.301761 28648 consensus_queue.cc:237] T 00000000000000000000000000000000 P 97282695dc384199a87add8d91dc30e9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "97282695dc384199a87add8d91dc30e9" member_type: VOTER last_known_addr { host: "127.10.198.62" port: 33447 } }
I20260925 22:48:37.302423 28643 sys_catalog.cc:564] T 00000000000000000000000000000000 P 97282695dc384199a87add8d91dc30e9 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 22:48:37.303604 28649 sys_catalog.cc:455] T 00000000000000000000000000000000 P 97282695dc384199a87add8d91dc30e9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "97282695dc384199a87add8d91dc30e9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "97282695dc384199a87add8d91dc30e9" member_type: VOTER last_known_addr { host: "127.10.198.62" port: 33447 } } }
I20260925 22:48:37.303640 28650 sys_catalog.cc:455] T 00000000000000000000000000000000 P 97282695dc384199a87add8d91dc30e9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 97282695dc384199a87add8d91dc30e9. Latest consensus state: current_term: 1 leader_uuid: "97282695dc384199a87add8d91dc30e9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "97282695dc384199a87add8d91dc30e9" member_type: VOTER last_known_addr { host: "127.10.198.62" port: 33447 } } }
I20260925 22:48:37.303752 28649 sys_catalog.cc:458] T 00000000000000000000000000000000 P 97282695dc384199a87add8d91dc30e9 [sys.catalog]: This master's current role is: LEADER
I20260925 22:48:37.303768 28650 sys_catalog.cc:458] T 00000000000000000000000000000000 P 97282695dc384199a87add8d91dc30e9 [sys.catalog]: This master's current role is: LEADER
W20260925 22:48:37.307919 28664 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 97282695dc384199a87add8d91dc30e9: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 22:48:37.308050 28664 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 22:48:37.308130 28657 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 22:48:37.308933 28657 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 22:48:37.316128 28657 catalog_manager.cc:1347] Generated new cluster ID: ea3c388563e44d03a4d13646d51a8ddf
I20260925 22:48:37.316247 28657 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 22:48:37.334722 28657 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 22:48:37.336236 28657 catalog_manager.cc:1504] Loading token signing keys...
I20260925 22:48:37.344652 28657 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 97282695dc384199a87add8d91dc30e9: Generated new TSK 0
I20260925 22:48:37.345611 28657 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 22:48:37.360045 11032 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrWXm8X/build/debug/bin/kudu
/tmp/dist-test-taskrWXm8X/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790376259650642-11032-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790376259650642-11032-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790376259650642-11032-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790376259650642-11032-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.198.1:0
--local_ip_for_outbound_sockets=127.10.198.1
--webserver_interface=127.10.198.1
--webserver_port=0
--tserver_master_addrs=127.10.198.62:33447
--builtin_ntp_servers=127.10.198.20:39847
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260925 22:48:37.483170 28667 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:48:37.483493 28667 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:48:37.483536 28667 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:48:37.488099 28667 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:48:37.488173 28667 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:48:37.488235 28667 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.198.1
I20260925 22:48:37.493392 28667 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.198.20:39847
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790376259650642-11032-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790376259650642-11032-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.198.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790376259650642-11032-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.198.1
--webserver_port=0
--tserver_master_addrs=127.10.198.62:33447
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28667
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.198.1
--log_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790376259650642-11032-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:09:16 UTC on ccycloud.quasar-afmhyw.root.comops.site
build id 4673379
I20260925 22:48:37.494540 28667 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:48:37.495962 28667 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:48:37.506826 28673 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:48:37.506994 28667 server_base.cc:1034] running on GCE node
W20260925 22:48:37.506837 28675 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:48:37.506815 28672 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:48:37.507673 28667 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:48:37.508512 28667 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:48:37.509737 28667 hybrid_clock.cc:648] HybridClock initialized: now 1790376517509706 us; error 56 us; skew 500 ppm
I20260925 22:48:37.512092 28667 webserver.cc:460] Webserver started at http://127.10.198.1:38697/ using document root <none> and password file <none>
I20260925 22:48:37.512943 28667 fs_manager.cc:359] Metadata directory not provided
I20260925 22:48:37.513048 28667 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:48:37.513336 28667 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:48:37.516646 28667 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790376259650642-11032-0/minicluster-data/ts-0/data/instance:
uuid: "6006642e51e249fd85b6a0ae68ac8fea"
format_stamp: "Formatted at 2026-09-25 22:48:37 on dist-test-slave-5vdr"
server_key: "c3dd565e955afde97b357ad6671b66a8"
server_key_iv: "ae074fc353c6577c09d5866c69fd80cd"
server_key_version: "encryptionkey@0"
I20260925 22:48:37.517427 28667 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790376259650642-11032-0/minicluster-data/ts-0/wal/instance:
uuid: "6006642e51e249fd85b6a0ae68ac8fea"
format_stamp: "Formatted at 2026-09-25 22:48:37 on dist-test-slave-5vdr"
server_key: "c3dd565e955afde97b357ad6671b66a8"
server_key_iv: "ae074fc353c6577c09d5866c69fd80cd"
server_key_version: "encryptionkey@0"
I20260925 22:48:37.524439 28667 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.006s sys 0.000s
I20260925 22:48:37.527346 28681 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:48:37.528867 28667 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 22:48:37.529004 28667 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790376259650642-11032-0/minicluster-data/ts-0/data,/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790376259650642-11032-0/minicluster-data/ts-0/wal
uuid: "6006642e51e249fd85b6a0ae68ac8fea"
format_stamp: "Formatted at 2026-09-25 22:48:37 on dist-test-slave-5vdr"
server_key: "c3dd565e955afde97b357ad6671b66a8"
server_key_iv: "ae074fc353c6577c09d5866c69fd80cd"
server_key_version: "encryptionkey@0"
I20260925 22:48:37.529137 28667 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790376259650642-11032-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790376259650642-11032-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790376259650642-11032-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:48:37.547410 28667 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:48:37.548208 28667 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:48:37.548381 28667 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:48:37.548952 28667 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 22:48:37.549958 28667 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:48:37.550010 28667 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:48:37.550055 28667 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:48:37.550076 28667 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 22:48:37.560012 28667 rpc_server.cc:307] RPC server started. Bound to: 127.10.198.1:43327
I20260925 22:48:37.560031 28794 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.198.1:43327 every 8 connection(s)
I20260925 22:48:37.561061 28667 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790376259650642-11032-0/minicluster-data/ts-0/data/info.pb
I20260925 22:48:37.561594 11032 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrWXm8X/build/debug/bin/kudu as pid 28667
I20260925 22:48:37.561725 11032 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790376259650642-11032-0/minicluster-data/ts-0/wal/instance
I20260925 22:48:37.562067 11032 external_mini_cluster.cc:1428] Setting key e9f77c74bf70d7c3511f50fc4d314c82
I20260925 22:48:37.564555 11032 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrWXm8X/build/debug/bin/kudu
/tmp/dist-test-taskrWXm8X/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790376259650642-11032-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790376259650642-11032-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790376259650642-11032-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790376259650642-11032-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.198.2:0
--local_ip_for_outbound_sockets=127.10.198.2
--webserver_interface=127.10.198.2
--webserver_port=0
--tserver_master_addrs=127.10.198.62:33447
--builtin_ntp_servers=127.10.198.20:39847
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260925 22:48:37.571957 28646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:48:37.563336 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.198.1:37241 (local address 127.10.198.62:33447)
0925 22:48:37.563483 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0925 22:48:37.563488 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:48:37.565183 (+ 1695us) server_negotiation.cc:407] Connection header received
0925 22:48:37.566692 (+ 1509us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:48:37.566696 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:48:37.566757 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:48:37.566851 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:48:37.567902 (+ 1051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:48:37.568460 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:48:37.569736 (+ 1276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:48:37.569971 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:48:37.570091 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:48:37.571376 (+ 1285us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:48:37.571379 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:48:37.571387 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 22:48:37.571402 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:48:37.571421 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 22:48:37.571430 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:48:37.571508 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0925 22:48:37.571681 (+ 173us) server_negotiation.cc:299] Negotiation successful
0925 22:48:37.571758 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260925 22:48:37.572909 28797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:48:37.564110 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.198.62:33447 (local address 127.10.198.1:37241)
0925 22:48:37.565024 (+ 914us) negotiation.cc:107] Waiting for socket to connect
0925 22:48:37.565073 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0925 22:48:37.566438 (+ 1365us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:48:37.567045 (+ 607us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:48:37.567054 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:48:37.567153 (+ 99us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:48:37.567710 (+ 557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:48:37.567725 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:48:37.568611 (+ 886us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:48:37.568621 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:48:37.569579 (+ 958us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:48:37.569591 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:48:37.570152 (+ 561us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:48:37.570156 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:48:37.570343 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:48:37.571046 (+ 703us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 22:48:37.571047 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:48:37.571094 (+ 47us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 22:48:37.571174 (+ 80us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:48:37.571510 (+ 336us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:48:37.571524 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:48:37.571532 (+ 8us) client_negotiation.cc:769] Sending connection context
0925 22:48:37.571678 (+ 146us) client_negotiation.cc:240] Negotiation successful
0925 22:48:37.572029 (+ 351us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":374,"thread_start_us":177,"threads_started":1}
I20260925 22:48:37.574396 28795 heartbeater.cc:347] Connected to a master server at 127.10.198.62:33447
I20260925 22:48:37.574682 28795 heartbeater.cc:464] Registering TS with master...
I20260925 22:48:37.575273 28795 heartbeater.cc:511] Master 127.10.198.62:33447 requested a full tablet report, sending...
I20260925 22:48:37.577219 28607 ts_manager.cc:194] Registered new tserver with Master: 6006642e51e249fd85b6a0ae68ac8fea (127.10.198.1:43327)
I20260925 22:48:37.578902 28607 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.10.198.1:37241
W20260925 22:48:37.687711 28798 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:48:37.688058 28798 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:48:37.688100 28798 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:48:37.692471 28798 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:48:37.692546 28798 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:48:37.692606 28798 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.198.2
I20260925 22:48:37.697714 28798 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.198.20:39847
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790376259650642-11032-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790376259650642-11032-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.198.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790376259650642-11032-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.198.2
--webserver_port=0
--tserver_master_addrs=127.10.198.62:33447
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28798
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.198.2
--log_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790376259650642-11032-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:09:16 UTC on ccycloud.quasar-afmhyw.root.comops.site
build id 4673379
I20260925 22:48:37.698868 28798 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:48:37.700371 28798 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:48:37.711392 28806 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:48:37.711400 28804 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:48:37.711392 28803 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:48:37.711791 28798 server_base.cc:1034] running on GCE node
I20260925 22:48:37.712394 28798 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:48:37.713126 28798 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:48:37.714336 28798 hybrid_clock.cc:648] HybridClock initialized: now 1790376517714317 us; error 39 us; skew 500 ppm
I20260925 22:48:37.716609 28798 webserver.cc:460] Webserver started at http://127.10.198.2:37501/ using document root <none> and password file <none>
I20260925 22:48:37.717248 28798 fs_manager.cc:359] Metadata directory not provided
I20260925 22:48:37.717357 28798 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:48:37.717599 28798 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:48:37.719834 28798 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790376259650642-11032-0/minicluster-data/ts-1/data/instance:
uuid: "f2c404c706e34e4ca69b966456dd3da3"
format_stamp: "Formatted at 2026-09-25 22:48:37 on dist-test-slave-5vdr"
server_key: "d676aafb453638089a6ab8295010d517"
server_key_iv: "dce07be79d46b9a133186db46094b07f"
server_key_version: "encryptionkey@0"
I20260925 22:48:37.720533 28798 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790376259650642-11032-0/minicluster-data/ts-1/wal/instance:
uuid: "f2c404c706e34e4ca69b966456dd3da3"
format_stamp: "Formatted at 2026-09-25 22:48:37 on dist-test-slave-5vdr"
server_key: "d676aafb453638089a6ab8295010d517"
server_key_iv: "dce07be79d46b9a133186db46094b07f"
server_key_version: "encryptionkey@0"
I20260925 22:48:37.724764 28798 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260925 22:48:37.727655 28812 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:48:37.729204 28798 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 22:48:37.729390 28798 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790376259650642-11032-0/minicluster-data/ts-1/data,/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790376259650642-11032-0/minicluster-data/ts-1/wal
uuid: "f2c404c706e34e4ca69b966456dd3da3"
format_stamp: "Formatted at 2026-09-25 22:48:37 on dist-test-slave-5vdr"
server_key: "d676aafb453638089a6ab8295010d517"
server_key_iv: "dce07be79d46b9a133186db46094b07f"
server_key_version: "encryptionkey@0"
I20260925 22:48:37.729534 28798 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790376259650642-11032-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790376259650642-11032-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790376259650642-11032-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:48:37.745349 28798 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:48:37.746119 28798 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:48:37.746344 28798 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:48:37.746960 28798 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 22:48:37.748108 28798 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:48:37.748176 28798 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:48:37.748250 28798 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:48:37.748293 28798 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 22:48:37.758040 28798 rpc_server.cc:307] RPC server started. Bound to: 127.10.198.2:38403
I20260925 22:48:37.758129 28925 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.198.2:38403 every 8 connection(s)
I20260925 22:48:37.759091 28798 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790376259650642-11032-0/minicluster-data/ts-1/data/info.pb
I20260925 22:48:37.761952 11032 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrWXm8X/build/debug/bin/kudu as pid 28798
I20260925 22:48:37.762115 11032 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790376259650642-11032-0/minicluster-data/ts-1/wal/instance
I20260925 22:48:37.762441 11032 external_mini_cluster.cc:1428] Setting key fc5c80d16f1c1222b04092037a3aff3d
I20260925 22:48:37.764758 11032 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrWXm8X/build/debug/bin/kudu
/tmp/dist-test-taskrWXm8X/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790376259650642-11032-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790376259650642-11032-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790376259650642-11032-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790376259650642-11032-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.198.3:0
--local_ip_for_outbound_sockets=127.10.198.3
--webserver_interface=127.10.198.3
--webserver_port=0
--tserver_master_addrs=127.10.198.62:33447
--builtin_ntp_servers=127.10.198.20:39847
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260925 22:48:37.771409 28646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:48:37.761290 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.198.2:47789 (local address 127.10.198.62:33447)
0925 22:48:37.761471 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0925 22:48:37.761475 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:48:37.762874 (+ 1399us) server_negotiation.cc:407] Connection header received
0925 22:48:37.764856 (+ 1982us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:48:37.764859 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:48:37.764922 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:48:37.764998 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:48:37.766223 (+ 1225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:48:37.766774 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:48:37.768928 (+ 2154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:48:37.769166 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:48:37.769262 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:48:37.770881 (+ 1619us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:48:37.770884 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:48:37.770887 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 22:48:37.770902 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:48:37.770922 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 22:48:37.770930 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:48:37.770992 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0925 22:48:37.771214 (+ 222us) server_negotiation.cc:299] Negotiation successful
0925 22:48:37.771283 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260925 22:48:37.772804 28928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:48:37.761781 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.198.62:33447 (local address 127.10.198.2:47789)
0925 22:48:37.762699 (+ 918us) negotiation.cc:107] Waiting for socket to connect
0925 22:48:37.762749 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0925 22:48:37.764596 (+ 1847us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:48:37.765267 (+ 671us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:48:37.765275 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:48:37.765378 (+ 103us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:48:37.766035 (+ 657us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:48:37.766061 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:48:37.766944 (+ 883us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:48:37.766950 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:48:37.768640 (+ 1690us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:48:37.768660 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:48:37.769378 (+ 718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:48:37.769385 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:48:37.769685 (+ 300us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:48:37.770472 (+ 787us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 22:48:37.770476 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:48:37.770549 (+ 73us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 22:48:37.770675 (+ 126us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:48:37.771023 (+ 348us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:48:37.771037 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:48:37.771047 (+ 10us) client_negotiation.cc:769] Sending connection context
0925 22:48:37.771216 (+ 169us) client_negotiation.cc:240] Negotiation successful
0925 22:48:37.771949 (+ 733us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":419,"thread_start_us":171,"threads_started":1}
I20260925 22:48:37.774169 28926 heartbeater.cc:347] Connected to a master server at 127.10.198.62:33447
I20260925 22:48:37.774502 28926 heartbeater.cc:464] Registering TS with master...
I20260925 22:48:37.775319 28926 heartbeater.cc:511] Master 127.10.198.62:33447 requested a full tablet report, sending...
I20260925 22:48:37.776755 28607 ts_manager.cc:194] Registered new tserver with Master: f2c404c706e34e4ca69b966456dd3da3 (127.10.198.2:38403)
I20260925 22:48:37.777381 28607 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.10.198.2:47789
W20260925 22:48:37.888125 28929 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:48:37.888445 28929 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:48:37.888486 28929 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:48:37.892843 28929 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:48:37.892920 28929 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:48:37.892977 28929 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.198.3
I20260925 22:48:37.898064 28929 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.198.20:39847
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790376259650642-11032-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790376259650642-11032-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.198.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790376259650642-11032-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.198.3
--webserver_port=0
--tserver_master_addrs=127.10.198.62:33447
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28929
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.198.3
--log_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790376259650642-11032-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:09:16 UTC on ccycloud.quasar-afmhyw.root.comops.site
build id 4673379
I20260925 22:48:37.899279 28929 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:48:37.900863 28929 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:48:37.911698 28937 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:48:37.911698 28935 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:48:37.911698 28934 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:48:37.912168 28929 server_base.cc:1034] running on GCE node
I20260925 22:48:37.912675 28929 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:48:37.913411 28929 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:48:37.914606 28929 hybrid_clock.cc:648] HybridClock initialized: now 1790376517914575 us; error 44 us; skew 500 ppm
I20260925 22:48:37.917375 28929 webserver.cc:460] Webserver started at http://127.10.198.3:35669/ using document root <none> and password file <none>
I20260925 22:48:37.918183 28929 fs_manager.cc:359] Metadata directory not provided
I20260925 22:48:37.918322 28929 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:48:37.918568 28929 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:48:37.921890 28929 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790376259650642-11032-0/minicluster-data/ts-2/data/instance:
uuid: "a08a008b1ad94c9d85197929b3679b1c"
format_stamp: "Formatted at 2026-09-25 22:48:37 on dist-test-slave-5vdr"
server_key: "c7799dc820a673a20a65405f578988f9"
server_key_iv: "4edc51b465b82cadf21ece35274e3347"
server_key_version: "encryptionkey@0"
I20260925 22:48:37.922833 28929 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790376259650642-11032-0/minicluster-data/ts-2/wal/instance:
uuid: "a08a008b1ad94c9d85197929b3679b1c"
format_stamp: "Formatted at 2026-09-25 22:48:37 on dist-test-slave-5vdr"
server_key: "c7799dc820a673a20a65405f578988f9"
server_key_iv: "4edc51b465b82cadf21ece35274e3347"
server_key_version: "encryptionkey@0"
I20260925 22:48:37.927711 28929 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260925 22:48:37.930702 28943 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:48:37.932246 28929 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 22:48:37.932399 28929 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790376259650642-11032-0/minicluster-data/ts-2/data,/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790376259650642-11032-0/minicluster-data/ts-2/wal
uuid: "a08a008b1ad94c9d85197929b3679b1c"
format_stamp: "Formatted at 2026-09-25 22:48:37 on dist-test-slave-5vdr"
server_key: "c7799dc820a673a20a65405f578988f9"
server_key_iv: "4edc51b465b82cadf21ece35274e3347"
server_key_version: "encryptionkey@0"
I20260925 22:48:37.932551 28929 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790376259650642-11032-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790376259650642-11032-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790376259650642-11032-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:48:37.947988 28929 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:48:37.948772 28929 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:48:37.948997 28929 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:48:37.949649 28929 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 22:48:37.950714 28929 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:48:37.950767 28929 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:48:37.950838 28929 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:48:37.950878 28929 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 22:48:37.960649 28929 rpc_server.cc:307] RPC server started. Bound to: 127.10.198.3:34091
I20260925 22:48:37.960680 29056 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.198.3:34091 every 8 connection(s)
I20260925 22:48:37.961704 28929 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790376259650642-11032-0/minicluster-data/ts-2/data/info.pb
I20260925 22:48:37.962592 11032 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrWXm8X/build/debug/bin/kudu as pid 28929
I20260925 22:48:37.962710 11032 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790376259650642-11032-0/minicluster-data/ts-2/wal/instance
I20260925 22:48:37.963002 11032 external_mini_cluster.cc:1428] Setting key ed53b7e20a8c5988204f6a757da3a2d3
I20260925 22:48:37.972471 28646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:48:37.963901 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.198.3:47995 (local address 127.10.198.62:33447)
0925 22:48:37.964061 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0925 22:48:37.964065 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:48:37.965578 (+ 1513us) server_negotiation.cc:407] Connection header received
0925 22:48:37.967112 (+ 1534us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:48:37.967116 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:48:37.967178 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:48:37.967259 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:48:37.968353 (+ 1094us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:48:37.968947 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:48:37.970246 (+ 1299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:48:37.970492 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:48:37.970604 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:48:37.971920 (+ 1316us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:48:37.971924 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:48:37.971926 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 22:48:37.971941 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:48:37.971961 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 22:48:37.971970 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:48:37.972049 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0925 22:48:37.972226 (+ 177us) server_negotiation.cc:299] Negotiation successful
0925 22:48:37.972313 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260925 22:48:37.973469 29059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:48:37.964539 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.198.62:33447 (local address 127.10.198.3:47995)
0925 22:48:37.965413 (+ 874us) negotiation.cc:107] Waiting for socket to connect
0925 22:48:37.965482 (+ 69us) client_negotiation.cc:174] Beginning negotiation
0925 22:48:37.966863 (+ 1381us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:48:37.967457 (+ 594us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:48:37.967465 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:48:37.967563 (+ 98us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:48:37.968188 (+ 625us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:48:37.968204 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:48:37.969110 (+ 906us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:48:37.969113 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:48:37.970092 (+ 979us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:48:37.970103 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:48:37.970674 (+ 571us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:48:37.970679 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:48:37.970899 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:48:37.971574 (+ 675us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 22:48:37.971575 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:48:37.971621 (+ 46us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 22:48:37.971698 (+ 77us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:48:37.972064 (+ 366us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:48:37.972077 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:48:37.972089 (+ 12us) client_negotiation.cc:769] Sending connection context
0925 22:48:37.972226 (+ 137us) client_negotiation.cc:240] Negotiation successful
0925 22:48:37.972562 (+ 336us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":316,"thread_start_us":130,"threads_started":1}
I20260925 22:48:37.974768 29057 heartbeater.cc:347] Connected to a master server at 127.10.198.62:33447
I20260925 22:48:37.975059 29057 heartbeater.cc:464] Registering TS with master...
I20260925 22:48:37.975668 29057 heartbeater.cc:511] Master 127.10.198.62:33447 requested a full tablet report, sending...
I20260925 22:48:37.976953 28607 ts_manager.cc:194] Registered new tserver with Master: a08a008b1ad94c9d85197929b3679b1c (127.10.198.3:34091)
I20260925 22:48:37.977523 11032 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 22:48:37.977638 28607 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.10.198.3:47995
I20260925 22:48:37.983691 28646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:48:37.979960 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58086 (local address 127.10.198.62:33447)
0925 22:48:37.980122 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0925 22:48:37.980128 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 22:48:37.980317 (+ 189us) server_negotiation.cc:407] Connection header received
0925 22:48:37.980408 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:48:37.980410 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:48:37.980450 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:48:37.980523 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:48:37.980800 (+ 277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:48:37.981330 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:48:37.982157 (+ 827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:48:37.982384 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:48:37.982466 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:48:37.983236 (+ 770us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:48:37.983239 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:48:37.983241 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 22:48:37.983255 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:48:37.983271 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 22:48:37.983278 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:48:37.983345 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0925 22:48:37.983429 (+ 84us) server_negotiation.cc:299] Negotiation successful
0925 22:48:37.983488 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260925 22:48:37.986821 28607 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58086:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 22:48:37.990073 28607 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 22:48:38.006645 29070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:48:37.999668 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.198.2:38403 (local address 127.0.0.1:48782)
0925 22:48:38.000515 (+ 847us) negotiation.cc:107] Waiting for socket to connect
0925 22:48:38.000530 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 22:48:38.000676 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:48:38.001671 (+ 995us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:48:38.001673 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:48:38.001691 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:48:38.001795 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:48:38.001802 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:48:38.003168 (+ 1366us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:48:38.003174 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:48:38.005131 (+ 1957us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:48:38.005139 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:48:38.006024 (+ 885us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:48:38.006028 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:48:38.006156 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:48:38.006196 (+ 40us) client_negotiation.cc:769] Sending connection context
0925 22:48:38.006317 (+ 121us) client_negotiation.cc:240] Negotiation successful
0925 22:48:38.006466 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":759,"mutex_wait_us":29,"spinlock_wait_cycles":2048,"thread_start_us":103,"threads_started":1}
I20260925 22:48:38.006647 29071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:48:38.000089 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48782 (local address 127.10.198.2:38403)
0925 22:48:38.001029 (+ 940us) server_negotiation.cc:206] Beginning negotiation
0925 22:48:38.001043 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0925 22:48:38.001068 (+ 25us) server_negotiation.cc:407] Connection header received
0925 22:48:38.001154 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:48:38.001159 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:48:38.001381 (+ 222us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:48:38.001528 (+ 147us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:48:38.001923 (+ 395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:48:38.003037 (+ 1114us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:48:38.005279 (+ 2242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:48:38.005860 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:48:38.006000 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:48:38.006214 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0925 22:48:38.006360 (+ 146us) server_negotiation.cc:299] Negotiation successful
0925 22:48:38.006468 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":252,"thread_start_us":135,"threads_started":1}
I20260925 22:48:38.007071 29068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:48:37.999435 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.198.1:43327 (local address 127.0.0.1:38918)
0925 22:48:37.999961 (+ 526us) negotiation.cc:107] Waiting for socket to connect
0925 22:48:38.000447 (+ 486us) client_negotiation.cc:174] Beginning negotiation
0925 22:48:38.000676 (+ 229us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:48:38.001464 (+ 788us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:48:38.001473 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:48:38.001500 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:48:38.001655 (+ 155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:48:38.001663 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:48:38.003316 (+ 1653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:48:38.003319 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:48:38.004833 (+ 1514us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:48:38.004845 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:48:38.006289 (+ 1444us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:48:38.006293 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:48:38.006438 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:48:38.006508 (+ 70us) client_negotiation.cc:769] Sending connection context
0925 22:48:38.006575 (+ 67us) client_negotiation.cc:240] Negotiation successful
0925 22:48:38.006934 (+ 359us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":273,"mutex_wait_us":448,"spinlock_wait_cycles":9600,"thread_start_us":139,"threads_started":1}
I20260925 22:48:38.007308 29069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:48:37.999566 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38918 (local address 127.10.198.1:43327)
0925 22:48:38.000125 (+ 559us) server_negotiation.cc:206] Beginning negotiation
0925 22:48:38.000138 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0925 22:48:38.000563 (+ 425us) server_negotiation.cc:407] Connection header received
0925 22:48:38.000901 (+ 338us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:48:38.000911 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:48:38.001165 (+ 254us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:48:38.001359 (+ 194us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:48:38.001796 (+ 437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:48:38.003152 (+ 1356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:48:38.004981 (+ 1829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:48:38.005862 (+ 881us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:48:38.005985 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:48:38.006215 (+ 230us) server_negotiation.cc:1035] Waiting for connection context
0925 22:48:38.006664 (+ 449us) server_negotiation.cc:299] Negotiation successful
0925 22:48:38.007150 (+ 486us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":453,"thread_start_us":151,"threads_started":1}
I20260925 22:48:38.007422 29073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:48:38.000758 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56638 (local address 127.10.198.3:34091)
0925 22:48:38.001379 (+ 621us) server_negotiation.cc:206] Beginning negotiation
0925 22:48:38.001391 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 22:48:38.001414 (+ 23us) server_negotiation.cc:407] Connection header received
0925 22:48:38.001490 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:48:38.001498 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:48:38.001652 (+ 154us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:48:38.001821 (+ 169us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:48:38.002008 (+ 187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:48:38.003161 (+ 1153us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:48:38.004849 (+ 1688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:48:38.005861 (+ 1012us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:48:38.005985 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:48:38.006215 (+ 230us) server_negotiation.cc:1035] Waiting for connection context
0925 22:48:38.007119 (+ 904us) server_negotiation.cc:299] Negotiation successful
0925 22:48:38.007240 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":498,"thread_start_us":165,"threads_started":1}
I20260925 22:48:38.007556 29072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:48:38.000351 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.198.3:34091 (local address 127.0.0.1:56638)
0925 22:48:38.001204 (+ 853us) negotiation.cc:107] Waiting for socket to connect
0925 22:48:38.001219 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 22:48:38.001314 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:48:38.001813 (+ 499us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:48:38.001815 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:48:38.001835 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:48:38.001919 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:48:38.001924 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:48:38.003315 (+ 1391us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:48:38.003319 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:48:38.004687 (+ 1368us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:48:38.004700 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:48:38.006302 (+ 1602us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:48:38.006306 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:48:38.006443 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:48:38.006498 (+ 55us) client_negotiation.cc:769] Sending connection context
0925 22:48:38.006575 (+ 77us) client_negotiation.cc:240] Negotiation successful
0925 22:48:38.006675 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":790,"mutex_wait_us":56,"thread_start_us":87,"threads_started":1}
I20260925 22:48:38.011164 28991 tablet_service.cc:1467] Processing CreateTablet for tablet 3d2c1e9d41fb47269e8447e7a858c070 (DEFAULT_TABLE table=test-table [id=cbd99af430684eee8bba991b4b30924d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 22:48:38.011178 28729 tablet_service.cc:1467] Processing CreateTablet for tablet 3d2c1e9d41fb47269e8447e7a858c070 (DEFAULT_TABLE table=test-table [id=cbd99af430684eee8bba991b4b30924d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 22:48:38.011849 28860 tablet_service.cc:1467] Processing CreateTablet for tablet 3d2c1e9d41fb47269e8447e7a858c070 (DEFAULT_TABLE table=test-table [id=cbd99af430684eee8bba991b4b30924d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 22:48:38.012538 28991 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3d2c1e9d41fb47269e8447e7a858c070. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:48:38.012538 28729 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3d2c1e9d41fb47269e8447e7a858c070. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:48:38.012898 28860 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3d2c1e9d41fb47269e8447e7a858c070. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:48:38.019613 29074 tablet_bootstrap.cc:492] T 3d2c1e9d41fb47269e8447e7a858c070 P f2c404c706e34e4ca69b966456dd3da3: Bootstrap starting.
I20260925 22:48:38.021293 29075 tablet_bootstrap.cc:492] T 3d2c1e9d41fb47269e8447e7a858c070 P a08a008b1ad94c9d85197929b3679b1c: Bootstrap starting.
I20260925 22:48:38.021536 29076 tablet_bootstrap.cc:492] T 3d2c1e9d41fb47269e8447e7a858c070 P 6006642e51e249fd85b6a0ae68ac8fea: Bootstrap starting.
I20260925 22:48:38.022648 29074 tablet_bootstrap.cc:654] T 3d2c1e9d41fb47269e8447e7a858c070 P f2c404c706e34e4ca69b966456dd3da3: Neither blocks nor log segments found. Creating new log.
I20260925 22:48:38.023597 29074 log.cc:824] T 3d2c1e9d41fb47269e8447e7a858c070 P f2c404c706e34e4ca69b966456dd3da3: Log is configured to *not* fsync() on all Append() calls
I20260925 22:48:38.023661 29075 tablet_bootstrap.cc:654] T 3d2c1e9d41fb47269e8447e7a858c070 P a08a008b1ad94c9d85197929b3679b1c: Neither blocks nor log segments found. Creating new log.
I20260925 22:48:38.023943 29076 tablet_bootstrap.cc:654] T 3d2c1e9d41fb47269e8447e7a858c070 P 6006642e51e249fd85b6a0ae68ac8fea: Neither blocks nor log segments found. Creating new log.
I20260925 22:48:38.024633 29075 log.cc:824] T 3d2c1e9d41fb47269e8447e7a858c070 P a08a008b1ad94c9d85197929b3679b1c: Log is configured to *not* fsync() on all Append() calls
I20260925 22:48:38.024719 29076 log.cc:824] T 3d2c1e9d41fb47269e8447e7a858c070 P 6006642e51e249fd85b6a0ae68ac8fea: Log is configured to *not* fsync() on all Append() calls
I20260925 22:48:38.026186 29074 tablet_bootstrap.cc:492] T 3d2c1e9d41fb47269e8447e7a858c070 P f2c404c706e34e4ca69b966456dd3da3: No bootstrap required, opened a new log
I20260925 22:48:38.026436 29074 ts_tablet_manager.cc:1397] T 3d2c1e9d41fb47269e8447e7a858c070 P f2c404c706e34e4ca69b966456dd3da3: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260925 22:48:38.026795 29075 tablet_bootstrap.cc:492] T 3d2c1e9d41fb47269e8447e7a858c070 P a08a008b1ad94c9d85197929b3679b1c: No bootstrap required, opened a new log
I20260925 22:48:38.027026 29075 ts_tablet_manager.cc:1397] T 3d2c1e9d41fb47269e8447e7a858c070 P a08a008b1ad94c9d85197929b3679b1c: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260925 22:48:38.027696 29076 tablet_bootstrap.cc:492] T 3d2c1e9d41fb47269e8447e7a858c070 P 6006642e51e249fd85b6a0ae68ac8fea: No bootstrap required, opened a new log
I20260925 22:48:38.027923 29076 ts_tablet_manager.cc:1397] T 3d2c1e9d41fb47269e8447e7a858c070 P 6006642e51e249fd85b6a0ae68ac8fea: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260925 22:48:38.031332 29076 raft_consensus.cc:348] T 3d2c1e9d41fb47269e8447e7a858c070 P 6006642e51e249fd85b6a0ae68ac8fea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6006642e51e249fd85b6a0ae68ac8fea" member_type: VOTER last_known_addr { host: "127.10.198.1" port: 43327 } } peers { permanent_uuid: "a08a008b1ad94c9d85197929b3679b1c" member_type: VOTER last_known_addr { host: "127.10.198.3" port: 34091 } } peers { permanent_uuid: "f2c404c706e34e4ca69b966456dd3da3" member_type: VOTER last_known_addr { host: "127.10.198.2" port: 38403 } }
I20260925 22:48:38.031350 29074 raft_consensus.cc:348] T 3d2c1e9d41fb47269e8447e7a858c070 P f2c404c706e34e4ca69b966456dd3da3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6006642e51e249fd85b6a0ae68ac8fea" member_type: VOTER last_known_addr { host: "127.10.198.1" port: 43327 } } peers { permanent_uuid: "a08a008b1ad94c9d85197929b3679b1c" member_type: VOTER last_known_addr { host: "127.10.198.3" port: 34091 } } peers { permanent_uuid: "f2c404c706e34e4ca69b966456dd3da3" member_type: VOTER last_known_addr { host: "127.10.198.2" port: 38403 } }
I20260925 22:48:38.031350 29075 raft_consensus.cc:348] T 3d2c1e9d41fb47269e8447e7a858c070 P a08a008b1ad94c9d85197929b3679b1c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6006642e51e249fd85b6a0ae68ac8fea" member_type: VOTER last_known_addr { host: "127.10.198.1" port: 43327 } } peers { permanent_uuid: "a08a008b1ad94c9d85197929b3679b1c" member_type: VOTER last_known_addr { host: "127.10.198.3" port: 34091 } } peers { permanent_uuid: "f2c404c706e34e4ca69b966456dd3da3" member_type: VOTER last_known_addr { host: "127.10.198.2" port: 38403 } }
I20260925 22:48:38.031631 29075 raft_consensus.cc:374] T 3d2c1e9d41fb47269e8447e7a858c070 P a08a008b1ad94c9d85197929b3679b1c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:48:38.031631 29074 raft_consensus.cc:374] T 3d2c1e9d41fb47269e8447e7a858c070 P f2c404c706e34e4ca69b966456dd3da3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:48:38.031641 29076 raft_consensus.cc:374] T 3d2c1e9d41fb47269e8447e7a858c070 P 6006642e51e249fd85b6a0ae68ac8fea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:48:38.031697 29076 raft_consensus.cc:729] T 3d2c1e9d41fb47269e8447e7a858c070 P 6006642e51e249fd85b6a0ae68ac8fea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6006642e51e249fd85b6a0ae68ac8fea, State: Initialized, Role: FOLLOWER
I20260925 22:48:38.031698 29075 raft_consensus.cc:729] T 3d2c1e9d41fb47269e8447e7a858c070 P a08a008b1ad94c9d85197929b3679b1c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a08a008b1ad94c9d85197929b3679b1c, State: Initialized, Role: FOLLOWER
I20260925 22:48:38.031708 29074 raft_consensus.cc:729] T 3d2c1e9d41fb47269e8447e7a858c070 P f2c404c706e34e4ca69b966456dd3da3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f2c404c706e34e4ca69b966456dd3da3, State: Initialized, Role: FOLLOWER
I20260925 22:48:38.032209 29076 consensus_queue.cc:260] T 3d2c1e9d41fb47269e8447e7a858c070 P 6006642e51e249fd85b6a0ae68ac8fea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6006642e51e249fd85b6a0ae68ac8fea" member_type: VOTER last_known_addr { host: "127.10.198.1" port: 43327 } } peers { permanent_uuid: "a08a008b1ad94c9d85197929b3679b1c" member_type: VOTER last_known_addr { host: "127.10.198.3" port: 34091 } } peers { permanent_uuid: "f2c404c706e34e4ca69b966456dd3da3" member_type: VOTER last_known_addr { host: "127.10.198.2" port: 38403 } }
I20260925 22:48:38.032222 29074 consensus_queue.cc:260] T 3d2c1e9d41fb47269e8447e7a858c070 P f2c404c706e34e4ca69b966456dd3da3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6006642e51e249fd85b6a0ae68ac8fea" member_type: VOTER last_known_addr { host: "127.10.198.1" port: 43327 } } peers { permanent_uuid: "a08a008b1ad94c9d85197929b3679b1c" member_type: VOTER last_known_addr { host: "127.10.198.3" port: 34091 } } peers { permanent_uuid: "f2c404c706e34e4ca69b966456dd3da3" member_type: VOTER last_known_addr { host: "127.10.198.2" port: 38403 } }
I20260925 22:48:38.032222 29075 consensus_queue.cc:260] T 3d2c1e9d41fb47269e8447e7a858c070 P a08a008b1ad94c9d85197929b3679b1c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6006642e51e249fd85b6a0ae68ac8fea" member_type: VOTER last_known_addr { host: "127.10.198.1" port: 43327 } } peers { permanent_uuid: "a08a008b1ad94c9d85197929b3679b1c" member_type: VOTER last_known_addr { host: "127.10.198.3" port: 34091 } } peers { permanent_uuid: "f2c404c706e34e4ca69b966456dd3da3" member_type: VOTER last_known_addr { host: "127.10.198.2" port: 38403 } }
I20260925 22:48:38.032966 28795 heartbeater.cc:503] Master 127.10.198.62:33447 was elected leader, sending a full tablet report...
I20260925 22:48:38.033363 29074 ts_tablet_manager.cc:1428] T 3d2c1e9d41fb47269e8447e7a858c070 P f2c404c706e34e4ca69b966456dd3da3: Time spent starting tablet: real 0.007s user 0.008s sys 0.000s
I20260925 22:48:38.033363 29075 ts_tablet_manager.cc:1428] T 3d2c1e9d41fb47269e8447e7a858c070 P a08a008b1ad94c9d85197929b3679b1c: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260925 22:48:38.033377 29076 ts_tablet_manager.cc:1428] T 3d2c1e9d41fb47269e8447e7a858c070 P 6006642e51e249fd85b6a0ae68ac8fea: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260925 22:48:38.033771 29057 heartbeater.cc:503] Master 127.10.198.62:33447 was elected leader, sending a full tablet report...
I20260925 22:48:38.034380 28926 heartbeater.cc:503] Master 127.10.198.62:33447 was elected leader, sending a full tablet report...
W20260925 22:48:38.062490 28796 tablet.cc:2369] T 3d2c1e9d41fb47269e8447e7a858c070 P 6006642e51e249fd85b6a0ae68ac8fea: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 22:48:38.064590 29082 raft_consensus.cc:482] T 3d2c1e9d41fb47269e8447e7a858c070 P f2c404c706e34e4ca69b966456dd3da3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 22:48:38.064803 29082 raft_consensus.cc:504] T 3d2c1e9d41fb47269e8447e7a858c070 P f2c404c706e34e4ca69b966456dd3da3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6006642e51e249fd85b6a0ae68ac8fea" member_type: VOTER last_known_addr { host: "127.10.198.1" port: 43327 } } peers { permanent_uuid: "a08a008b1ad94c9d85197929b3679b1c" member_type: VOTER last_known_addr { host: "127.10.198.3" port: 34091 } } peers { permanent_uuid: "f2c404c706e34e4ca69b966456dd3da3" member_type: VOTER last_known_addr { host: "127.10.198.2" port: 38403 } }
I20260925 22:48:38.065963 29082 leader_election.cc:288] T 3d2c1e9d41fb47269e8447e7a858c070 P f2c404c706e34e4ca69b966456dd3da3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6006642e51e249fd85b6a0ae68ac8fea (127.10.198.1:43327), a08a008b1ad94c9d85197929b3679b1c (127.10.198.3:34091)
I20260925 22:48:38.070456 29069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:48:38.066407 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.198.2:35581 (local address 127.10.198.1:43327)
0925 22:48:38.066550 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0925 22:48:38.066555 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:48:38.066570 (+ 15us) server_negotiation.cc:407] Connection header received
0925 22:48:38.066683 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:48:38.066689 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:48:38.066744 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:48:38.067185 (+ 441us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:48:38.067323 (+ 138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:48:38.068136 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:48:38.069425 (+ 1289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:48:38.069861 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:48:38.069998 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:48:38.070051 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0925 22:48:38.070273 (+ 222us) server_negotiation.cc:299] Negotiation successful
0925 22:48:38.070344 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260925 22:48:38.070456 29073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:48:38.066738 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.198.2:57801 (local address 127.10.198.3:34091)
0925 22:48:38.066895 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0925 22:48:38.066900 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:48:38.067048 (+ 148us) server_negotiation.cc:407] Connection header received
0925 22:48:38.067147 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:48:38.067153 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:48:38.067206 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:48:38.067284 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:48:38.067532 (+ 248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:48:38.068269 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:48:38.069486 (+ 1217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:48:38.069871 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:48:38.070022 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:48:38.070075 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0925 22:48:38.070240 (+ 165us) server_negotiation.cc:299] Negotiation successful
0925 22:48:38.070325 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260925 22:48:38.070456 28928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:48:38.066298 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.198.1:43327 (local address 127.10.198.2:35581)
0925 22:48:38.066435 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0925 22:48:38.066468 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0925 22:48:38.066581 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:48:38.066886 (+ 305us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:48:38.066891 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:48:38.066915 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:48:38.067062 (+ 147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:48:38.067073 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:48:38.068300 (+ 1227us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:48:38.068303 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:48:38.069280 (+ 977us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:48:38.069299 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:48:38.070022 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:48:38.070026 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:48:38.070126 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:48:38.070146 (+ 20us) client_negotiation.cc:769] Sending connection context
0925 22:48:38.070242 (+ 96us) client_negotiation.cc:240] Negotiation successful
0925 22:48:38.070317 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":51,"mutex_wait_us":1}
I20260925 22:48:38.070470 29083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:48:38.066629 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.198.3:34091 (local address 127.10.198.2:57801)
0925 22:48:38.066954 (+ 325us) negotiation.cc:107] Waiting for socket to connect
0925 22:48:38.066968 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 22:48:38.067062 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:48:38.067309 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:48:38.067311 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:48:38.067324 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:48:38.067428 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:48:38.067432 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:48:38.068365 (+ 933us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:48:38.068370 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:48:38.069368 (+ 998us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:48:38.069382 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:48:38.069998 (+ 616us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:48:38.070002 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:48:38.070103 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:48:38.070126 (+ 23us) client_negotiation.cc:769] Sending connection context
0925 22:48:38.070195 (+ 69us) client_negotiation.cc:240] Negotiation successful
0925 22:48:38.070297 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":247,"thread_start_us":151,"threads_started":1}
I20260925 22:48:38.071204 28749 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d2c1e9d41fb47269e8447e7a858c070" candidate_uuid: "f2c404c706e34e4ca69b966456dd3da3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6006642e51e249fd85b6a0ae68ac8fea" is_pre_election: true
I20260925 22:48:38.071388 29011 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d2c1e9d41fb47269e8447e7a858c070" candidate_uuid: "f2c404c706e34e4ca69b966456dd3da3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a08a008b1ad94c9d85197929b3679b1c" is_pre_election: true
I20260925 22:48:38.071563 28749 raft_consensus.cc:2454] T 3d2c1e9d41fb47269e8447e7a858c070 P 6006642e51e249fd85b6a0ae68ac8fea [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f2c404c706e34e4ca69b966456dd3da3 in term 0.
I20260925 22:48:38.071710 29011 raft_consensus.cc:2454] T 3d2c1e9d41fb47269e8447e7a858c070 P a08a008b1ad94c9d85197929b3679b1c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f2c404c706e34e4ca69b966456dd3da3 in term 0.
I20260925 22:48:38.072145 28813 leader_election.cc:302] T 3d2c1e9d41fb47269e8447e7a858c070 P f2c404c706e34e4ca69b966456dd3da3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6006642e51e249fd85b6a0ae68ac8fea, f2c404c706e34e4ca69b966456dd3da3; no voters:
I20260925 22:48:38.072444 29082 raft_consensus.cc:2781] T 3d2c1e9d41fb47269e8447e7a858c070 P f2c404c706e34e4ca69b966456dd3da3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 22:48:38.072539 29082 raft_consensus.cc:482] T 3d2c1e9d41fb47269e8447e7a858c070 P f2c404c706e34e4ca69b966456dd3da3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 22:48:38.072585 29082 raft_consensus.cc:3037] T 3d2c1e9d41fb47269e8447e7a858c070 P f2c404c706e34e4ca69b966456dd3da3 [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:48:38.073671 29082 raft_consensus.cc:504] T 3d2c1e9d41fb47269e8447e7a858c070 P f2c404c706e34e4ca69b966456dd3da3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6006642e51e249fd85b6a0ae68ac8fea" member_type: VOTER last_known_addr { host: "127.10.198.1" port: 43327 } } peers { permanent_uuid: "a08a008b1ad94c9d85197929b3679b1c" member_type: VOTER last_known_addr { host: "127.10.198.3" port: 34091 } } peers { permanent_uuid: "f2c404c706e34e4ca69b966456dd3da3" member_type: VOTER last_known_addr { host: "127.10.198.2" port: 38403 } }
I20260925 22:48:38.074083 29082 leader_election.cc:288] T 3d2c1e9d41fb47269e8447e7a858c070 P f2c404c706e34e4ca69b966456dd3da3 [CANDIDATE]: Term 1 election: Requested vote from peers 6006642e51e249fd85b6a0ae68ac8fea (127.10.198.1:43327), a08a008b1ad94c9d85197929b3679b1c (127.10.198.3:34091)
I20260925 22:48:38.074543 29011 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d2c1e9d41fb47269e8447e7a858c070" candidate_uuid: "f2c404c706e34e4ca69b966456dd3da3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a08a008b1ad94c9d85197929b3679b1c"
I20260925 22:48:38.074536 28749 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d2c1e9d41fb47269e8447e7a858c070" candidate_uuid: "f2c404c706e34e4ca69b966456dd3da3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6006642e51e249fd85b6a0ae68ac8fea"
I20260925 22:48:38.074680 29011 raft_consensus.cc:3037] T 3d2c1e9d41fb47269e8447e7a858c070 P a08a008b1ad94c9d85197929b3679b1c [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:48:38.074680 28749 raft_consensus.cc:3037] T 3d2c1e9d41fb47269e8447e7a858c070 P 6006642e51e249fd85b6a0ae68ac8fea [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:48:38.076089 29011 raft_consensus.cc:2454] T 3d2c1e9d41fb47269e8447e7a858c070 P a08a008b1ad94c9d85197929b3679b1c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f2c404c706e34e4ca69b966456dd3da3 in term 1.
I20260925 22:48:38.076089 28749 raft_consensus.cc:2454] T 3d2c1e9d41fb47269e8447e7a858c070 P 6006642e51e249fd85b6a0ae68ac8fea [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f2c404c706e34e4ca69b966456dd3da3 in term 1.
I20260925 22:48:38.076463 28813 leader_election.cc:302] T 3d2c1e9d41fb47269e8447e7a858c070 P f2c404c706e34e4ca69b966456dd3da3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6006642e51e249fd85b6a0ae68ac8fea, f2c404c706e34e4ca69b966456dd3da3; no voters:
I20260925 22:48:38.076661 29082 raft_consensus.cc:2781] T 3d2c1e9d41fb47269e8447e7a858c070 P f2c404c706e34e4ca69b966456dd3da3 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 22:48:38.076905 29082 raft_consensus.cc:686] T 3d2c1e9d41fb47269e8447e7a858c070 P f2c404c706e34e4ca69b966456dd3da3 [term 1 LEADER]: Becoming Leader. State: Replica: f2c404c706e34e4ca69b966456dd3da3, State: Running, Role: LEADER
I20260925 22:48:38.077250 29082 consensus_queue.cc:237] T 3d2c1e9d41fb47269e8447e7a858c070 P f2c404c706e34e4ca69b966456dd3da3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6006642e51e249fd85b6a0ae68ac8fea" member_type: VOTER last_known_addr { host: "127.10.198.1" port: 43327 } } peers { permanent_uuid: "a08a008b1ad94c9d85197929b3679b1c" member_type: VOTER last_known_addr { host: "127.10.198.3" port: 34091 } } peers { permanent_uuid: "f2c404c706e34e4ca69b966456dd3da3" member_type: VOTER last_known_addr { host: "127.10.198.2" port: 38403 } }
I20260925 22:48:38.080857 28604 catalog_manager.cc:5495] T 3d2c1e9d41fb47269e8447e7a858c070 P f2c404c706e34e4ca69b966456dd3da3 reported cstate change: term changed from 0 to 1, leader changed from <none> to f2c404c706e34e4ca69b966456dd3da3 (127.10.198.2). New cstate: current_term: 1 leader_uuid: "f2c404c706e34e4ca69b966456dd3da3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6006642e51e249fd85b6a0ae68ac8fea" member_type: VOTER last_known_addr { host: "127.10.198.1" port: 43327 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a08a008b1ad94c9d85197929b3679b1c" member_type: VOTER last_known_addr { host: "127.10.198.3" port: 34091 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f2c404c706e34e4ca69b966456dd3da3" member_type: VOTER last_known_addr { host: "127.10.198.2" port: 38403 } health_report { overall_health: HEALTHY } } }
I20260925 22:48:38.092587 29071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:48:38.089219 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48784 (local address 127.10.198.2:38403)
0925 22:48:38.089363 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0925 22:48:38.089367 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:48:38.089380 (+ 13us) server_negotiation.cc:407] Connection header received
0925 22:48:38.089458 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:48:38.089460 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:48:38.089517 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:48:38.089608 (+ 91us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 22:48:38.089817 (+ 209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:48:38.090319 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:48:38.091139 (+ 820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:48:38.091363 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:48:38.091440 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:48:38.091702 (+ 262us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 22:48:38.092220 (+ 518us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 22:48:38.092332 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0925 22:48:38.092398 (+ 66us) server_negotiation.cc:299] Negotiation successful
0925 22:48:38.092449 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260925 22:48:38.099104 28749 raft_consensus.cc:1261] T 3d2c1e9d41fb47269e8447e7a858c070 P 6006642e51e249fd85b6a0ae68ac8fea [term 1 FOLLOWER]: Refusing update from remote peer f2c404c706e34e4ca69b966456dd3da3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 22:48:38.099104 29011 raft_consensus.cc:1261] T 3d2c1e9d41fb47269e8447e7a858c070 P a08a008b1ad94c9d85197929b3679b1c [term 1 FOLLOWER]: Refusing update from remote peer f2c404c706e34e4ca69b966456dd3da3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 22:48:38.100039 29084 consensus_queue.cc:1035] T 3d2c1e9d41fb47269e8447e7a858c070 P f2c404c706e34e4ca69b966456dd3da3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6006642e51e249fd85b6a0ae68ac8fea" member_type: VOTER last_known_addr { host: "127.10.198.1" port: 43327 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 22:48:38.100294 29082 consensus_queue.cc:1035] T 3d2c1e9d41fb47269e8447e7a858c070 P f2c404c706e34e4ca69b966456dd3da3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a08a008b1ad94c9d85197929b3679b1c" member_type: VOTER last_known_addr { host: "127.10.198.3" port: 34091 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 22:48:38.109740 29088 mvcc.cc:205] Tried to move back new op lower bound from 7333382218125373440 to 7333382218047217664. Current Snapshot: MvccSnapshot[applied={T|T < 7333382218125373440}]
I20260925 22:48:38.110591 29089 mvcc.cc:205] Tried to move back new op lower bound from 7333382218125373440 to 7333382218047217664. Current Snapshot: MvccSnapshot[applied={T|T < 7333382218125373440}]
I20260925 22:48:38.118986 28604 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:58086:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 22:48:38.119206 28604 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:58086:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 22:48:38.122071 28604 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 97282695dc384199a87add8d91dc30e9: Sending DeleteTablet for 3 replicas of tablet 3d2c1e9d41fb47269e8447e7a858c070
I20260925 22:48:38.123032 28860 tablet_service.cc:1514] Processing DeleteTablet for tablet 3d2c1e9d41fb47269e8447e7a858c070 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 22:48:38 UTC) from {username='slave'} at 127.0.0.1:48782
I20260925 22:48:38.123265 28991 tablet_service.cc:1514] Processing DeleteTablet for tablet 3d2c1e9d41fb47269e8447e7a858c070 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 22:48:38 UTC) from {username='slave'} at 127.0.0.1:56638
I20260925 22:48:38.123642 28729 tablet_service.cc:1514] Processing DeleteTablet for tablet 3d2c1e9d41fb47269e8447e7a858c070 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 22:48:38 UTC) from {username='slave'} at 127.0.0.1:38918
I20260925 22:48:38.124099 11032 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrWXm8X/build/debug/bin/kudu with pid 28667
I20260925 22:48:38.125241 29097 tablet_replica.cc:330] T 3d2c1e9d41fb47269e8447e7a858c070 P f2c404c706e34e4ca69b966456dd3da3: stopping tablet replica
I20260925 22:48:38.125715 29097 raft_consensus.cc:2229] T 3d2c1e9d41fb47269e8447e7a858c070 P f2c404c706e34e4ca69b966456dd3da3 [term 1 LEADER]: Raft consensus shutting down.
I20260925 22:48:38.126281 29097 raft_consensus.cc:2258] T 3d2c1e9d41fb47269e8447e7a858c070 P f2c404c706e34e4ca69b966456dd3da3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 22:48:38.128253 29097 ts_tablet_manager.cc:1905] T 3d2c1e9d41fb47269e8447e7a858c070 P f2c404c706e34e4ca69b966456dd3da3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 22:48:38.131361 29098 tablet_replica.cc:330] T 3d2c1e9d41fb47269e8447e7a858c070 P a08a008b1ad94c9d85197929b3679b1c: stopping tablet replica
I20260925 22:48:38.131870 29098 raft_consensus.cc:2229] T 3d2c1e9d41fb47269e8447e7a858c070 P a08a008b1ad94c9d85197929b3679b1c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 22:48:38.132105 29097 ts_tablet_manager.cc:1918] T 3d2c1e9d41fb47269e8447e7a858c070 P f2c404c706e34e4ca69b966456dd3da3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 22:48:38.132232 29097 log.cc:1196] T 3d2c1e9d41fb47269e8447e7a858c070 P f2c404c706e34e4ca69b966456dd3da3: Deleting WAL directory at /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790376259650642-11032-0/minicluster-data/ts-1/wal/wals/3d2c1e9d41fb47269e8447e7a858c070
W20260925 22:48:38.132392 28591 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.10.198.1:43327 (error 108)
I20260925 22:48:38.132443 29098 raft_consensus.cc:2258] T 3d2c1e9d41fb47269e8447e7a858c070 P a08a008b1ad94c9d85197929b3679b1c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 22:48:38.132611 29097 ts_tablet_manager.cc:1939] T 3d2c1e9d41fb47269e8447e7a858c070 P f2c404c706e34e4ca69b966456dd3da3: Deleting consensus metadata
I20260925 22:48:38.133307 11032 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrWXm8X/build/debug/bin/kudu with pid 28798
I20260925 22:48:38.134629 29098 ts_tablet_manager.cc:1905] T 3d2c1e9d41fb47269e8447e7a858c070 P a08a008b1ad94c9d85197929b3679b1c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 22:48:38.135704 29068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:48:38.135344 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.198.1:43327 (local address 127.0.0.1:38928)
0925 22:48:38.135519 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0925 22:48:38.135605 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.198.1:43327: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
W20260925 22:48:38.135962 28591 catalog_manager.cc:4584] TS 6006642e51e249fd85b6a0ae68ac8fea (127.10.198.1:43327): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 3d2c1e9d41fb47269e8447e7a858c070: Network error: Client connection negotiation failed: client connection to 127.10.198.1:43327: connect: Connection refused (error 111)
I20260925 22:48:38.137667 29098 ts_tablet_manager.cc:1918] T 3d2c1e9d41fb47269e8447e7a858c070 P a08a008b1ad94c9d85197929b3679b1c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 22:48:38.137771 29098 log.cc:1196] T 3d2c1e9d41fb47269e8447e7a858c070 P a08a008b1ad94c9d85197929b3679b1c: Deleting WAL directory at /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790376259650642-11032-0/minicluster-data/ts-2/wal/wals/3d2c1e9d41fb47269e8447e7a858c070
I20260925 22:48:38.138063 29098 ts_tablet_manager.cc:1939] T 3d2c1e9d41fb47269e8447e7a858c070 P a08a008b1ad94c9d85197929b3679b1c: Deleting consensus metadata
I20260925 22:48:38.139422 28591 catalog_manager.cc:4841] TS a08a008b1ad94c9d85197929b3679b1c (127.10.198.3:34091): tablet 3d2c1e9d41fb47269e8447e7a858c070 (table test-table [id=cbd99af430684eee8bba991b4b30924d]) successfully deleted
I20260925 22:48:38.140805 11032 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrWXm8X/build/debug/bin/kudu with pid 28929
I20260925 22:48:38.141731 29068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:48:38.140995 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.198.2:38403 (local address 127.0.0.1:48786)
0925 22:48:38.141524 (+ 529us) negotiation.cc:107] Waiting for socket to connect
0925 22:48:38.141597 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.198.2:38403: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":424}
I20260925 22:48:38.146907 11032 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrWXm8X/build/debug/bin/kudu with pid 28576
2026-09-25T22:48:38Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (1091 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790376259650642-11032-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 22:48:38 dist-test-slave-5vdr krb5kdc[29103](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 22:48:38 dist-test-slave-5vdr krb5kdc[29103](info): set up 2 sockets
Sep 25 22:48:38 dist-test-slave-5vdr krb5kdc[29103](info): commencing operation
krb5kdc: starting...
W20260925 22:48:39.216233 11032 mini_kdc.cc:121] Time spent starting KDC: real 1.047s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790376259650642-11032-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790376259650642-11032-0/krb5kdc/test-user.keytab.
Sep 25 22:48:39 dist-test-slave-5vdr krb5kdc[29103](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376519, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T22:48:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T22:48:39Z Disabled control of system clock
WARNING: no policy specified for kudu/127.10.198.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.198.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.198.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790376259650642-11032-0/krb5kdc/kudu_127.10.198.62.keytab.
Entry for principal kudu/127.10.198.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790376259650642-11032-0/krb5kdc/kudu_127.10.198.62.keytab.
I20260925 22:48:39.351466 11032 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrWXm8X/build/debug/bin/kudu
/tmp/dist-test-taskrWXm8X/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790376259650642-11032-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790376259650642-11032-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790376259650642-11032-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790376259650642-11032-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.198.62:38513
--webserver_interface=127.10.198.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790376259650642-11032-0/krb5kdc/kudu_127.10.198.62.keytab
--principal=kudu/127.10.198.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.198.20:44873
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.198.62:38513
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790376259650642-11032-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790376259650642-11032-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790376259650642-11032-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:48:39.472528 29116 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:48:39.472894 29116 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:48:39.472990 29116 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:48:39.477427 29116 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 22:48:39.477552 29116 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:48:39.477591 29116 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:48:39.477643 29116 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 22:48:39.477706 29116 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 22:48:39.483629 29116 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.198.20:44873
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790376259650642-11032-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790376259650642-11032-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.198.62:38513
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790376259650642-11032-0/krb5kdc/kudu_127.10.198.62.keytab
--principal=kudu/127.10.198.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.198.62:38513
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790376259650642-11032-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.198.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29116
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790376259650642-11032-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:09:16 UTC on ccycloud.quasar-afmhyw.root.comops.site
build id 4673379
I20260925 22:48:39.484958 29116 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:48:39.486125 29116 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:48:39.495741 29121 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:48:39.495798 29122 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:48:39.495801 29124 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:48:39.496212 29116 server_base.cc:1034] running on GCE node
I20260925 22:48:39.497076 29116 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:48:39.498556 29116 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:48:39.499801 29116 hybrid_clock.cc:648] HybridClock initialized: now 1790376519499770 us; error 53 us; skew 500 ppm
Sep 25 22:48:39 dist-test-slave-5vdr krb5kdc[29103](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376519, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.198.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:48:39.503284 29116 init.cc:377] Logged in from keytab as kudu/127.10.198.62@KRBTEST.COM (short username kudu)
I20260925 22:48:39.504482 29116 webserver.cc:460] Webserver started at http://127.10.198.62:38759/ using document root <none> and password file <none>
I20260925 22:48:39.505147 29116 fs_manager.cc:359] Metadata directory not provided
I20260925 22:48:39.505257 29116 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:48:39.505514 29116 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:48:39.507906 29116 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790376259650642-11032-0/minicluster-data/master-0/data/instance:
uuid: "e452e50fa17e4ebb948423eb256aeeb5"
format_stamp: "Formatted at 2026-09-25 22:48:39 on dist-test-slave-5vdr"
server_key: "e096a1934236c0214cb86323fa2e9b3a"
server_key_iv: "de54243fe7282c023623f8fc21ae2878"
server_key_version: "encryptionkey@0"
I20260925 22:48:39.508643 29116 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790376259650642-11032-0/minicluster-data/master-0/wal/instance:
uuid: "e452e50fa17e4ebb948423eb256aeeb5"
format_stamp: "Formatted at 2026-09-25 22:48:39 on dist-test-slave-5vdr"
server_key: "e096a1934236c0214cb86323fa2e9b3a"
server_key_iv: "de54243fe7282c023623f8fc21ae2878"
server_key_version: "encryptionkey@0"
I20260925 22:48:39.513563 29116 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260925 22:48:39.517943 29131 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:48:39.519778 29116 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260925 22:48:39.520033 29116 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790376259650642-11032-0/minicluster-data/master-0/data,/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790376259650642-11032-0/minicluster-data/master-0/wal
uuid: "e452e50fa17e4ebb948423eb256aeeb5"
format_stamp: "Formatted at 2026-09-25 22:48:39 on dist-test-slave-5vdr"
server_key: "e096a1934236c0214cb86323fa2e9b3a"
server_key_iv: "de54243fe7282c023623f8fc21ae2878"
server_key_version: "encryptionkey@0"
I20260925 22:48:39.520187 29116 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790376259650642-11032-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790376259650642-11032-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790376259650642-11032-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:48:39.540093 29116 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:48:39.543395 29116 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:48:39.543574 29116 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:48:39.551743 29116 rpc_server.cc:307] RPC server started. Bound to: 127.10.198.62:38513
I20260925 22:48:39.551752 29183 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.198.62:38513 every 8 connection(s)
I20260925 22:48:39.552908 29116 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790376259650642-11032-0/minicluster-data/master-0/data/info.pb
I20260925 22:48:39.556162 29184 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:48:39.559151 11032 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrWXm8X/build/debug/bin/kudu as pid 29116
I20260925 22:48:39.559352 11032 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790376259650642-11032-0/minicluster-data/master-0/wal/instance
I20260925 22:48:39.559680 11032 external_mini_cluster.cc:1428] Setting key cabc8bb9681cea0b66924909d004b110
I20260925 22:48:39.563117 29184 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e452e50fa17e4ebb948423eb256aeeb5: Bootstrap starting.
I20260925 22:48:39.565735 29184 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e452e50fa17e4ebb948423eb256aeeb5: Neither blocks nor log segments found. Creating new log.
I20260925 22:48:39.566629 29184 log.cc:824] T 00000000000000000000000000000000 P e452e50fa17e4ebb948423eb256aeeb5: Log is configured to *not* fsync() on all Append() calls
I20260925 22:48:39.569121 29184 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e452e50fa17e4ebb948423eb256aeeb5: No bootstrap required, opened a new log
Sep 25 22:48:39 dist-test-slave-5vdr krb5kdc[29103](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376519, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.198.62@KRBTEST.COM
I20260925 22:48:39.573205 29184 raft_consensus.cc:348] T 00000000000000000000000000000000 P e452e50fa17e4ebb948423eb256aeeb5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e452e50fa17e4ebb948423eb256aeeb5" member_type: VOTER last_known_addr { host: "127.10.198.62" port: 38513 } }
I20260925 22:48:39.573623 29184 raft_consensus.cc:374] T 00000000000000000000000000000000 P e452e50fa17e4ebb948423eb256aeeb5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:48:39.573683 29184 raft_consensus.cc:729] T 00000000000000000000000000000000 P e452e50fa17e4ebb948423eb256aeeb5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e452e50fa17e4ebb948423eb256aeeb5, State: Initialized, Role: FOLLOWER
I20260925 22:48:39.574282 29184 consensus_queue.cc:260] T 00000000000000000000000000000000 P e452e50fa17e4ebb948423eb256aeeb5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e452e50fa17e4ebb948423eb256aeeb5" member_type: VOTER last_known_addr { host: "127.10.198.62" port: 38513 } }
I20260925 22:48:39.574501 29184 raft_consensus.cc:388] T 00000000000000000000000000000000 P e452e50fa17e4ebb948423eb256aeeb5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 22:48:39.574576 29184 raft_consensus.cc:482] T 00000000000000000000000000000000 P e452e50fa17e4ebb948423eb256aeeb5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 22:48:39.574685 29184 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e452e50fa17e4ebb948423eb256aeeb5 [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:48:39.576110 29184 raft_consensus.cc:504] T 00000000000000000000000000000000 P e452e50fa17e4ebb948423eb256aeeb5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e452e50fa17e4ebb948423eb256aeeb5" member_type: VOTER last_known_addr { host: "127.10.198.62" port: 38513 } }
I20260925 22:48:39.576538 29184 leader_election.cc:302] T 00000000000000000000000000000000 P e452e50fa17e4ebb948423eb256aeeb5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e452e50fa17e4ebb948423eb256aeeb5; no voters:
I20260925 22:48:39.576951 29184 leader_election.cc:288] T 00000000000000000000000000000000 P e452e50fa17e4ebb948423eb256aeeb5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 22:48:39.577105 29189 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e452e50fa17e4ebb948423eb256aeeb5 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 22:48:39.577116 29187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:48:39.561229 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36304 (local address 127.10.198.62:38513)
0925 22:48:39.561756 (+ 527us) server_negotiation.cc:206] Beginning negotiation
0925 22:48:39.561772 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0925 22:48:39.561812 (+ 40us) server_negotiation.cc:407] Connection header received
0925 22:48:39.562497 (+ 685us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:48:39.562518 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:48:39.562891 (+ 373us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:48:39.563272 (+ 381us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:48:39.564365 (+ 1093us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:48:39.565415 (+ 1050us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:48:39.566428 (+ 1013us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:48:39.567165 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:48:39.567365 (+ 200us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:48:39.570439 (+ 3074us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:48:39.570445 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:48:39.570464 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:48:39.570501 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:48:39.573135 (+ 2634us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:48:39.573837 (+ 702us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:48:39.573844 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:48:39.573851 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:48:39.574022 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:48:39.574402 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:48:39.574406 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:48:39.574407 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:48:39.575047 (+ 640us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:48:39.575267 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0925 22:48:39.575602 (+ 335us) server_negotiation.cc:299] Negotiation successful
0925 22:48:39.576003 (+ 401us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":365,"thread_start_us":147,"threads_started":1}
I20260925 22:48:39.577395 29189 raft_consensus.cc:686] T 00000000000000000000000000000000 P e452e50fa17e4ebb948423eb256aeeb5 [term 1 LEADER]: Becoming Leader. State: Replica: e452e50fa17e4ebb948423eb256aeeb5, State: Running, Role: LEADER
I20260925 22:48:39.577787 29189 consensus_queue.cc:237] T 00000000000000000000000000000000 P e452e50fa17e4ebb948423eb256aeeb5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e452e50fa17e4ebb948423eb256aeeb5" member_type: VOTER last_known_addr { host: "127.10.198.62" port: 38513 } }
I20260925 22:48:39.578343 29184 sys_catalog.cc:564] T 00000000000000000000000000000000 P e452e50fa17e4ebb948423eb256aeeb5 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 22:48:39.579079 29191 sys_catalog.cc:455] T 00000000000000000000000000000000 P e452e50fa17e4ebb948423eb256aeeb5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e452e50fa17e4ebb948423eb256aeeb5. Latest consensus state: current_term: 1 leader_uuid: "e452e50fa17e4ebb948423eb256aeeb5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e452e50fa17e4ebb948423eb256aeeb5" member_type: VOTER last_known_addr { host: "127.10.198.62" port: 38513 } } }
I20260925 22:48:39.579205 29191 sys_catalog.cc:458] T 00000000000000000000000000000000 P e452e50fa17e4ebb948423eb256aeeb5 [sys.catalog]: This master's current role is: LEADER
I20260925 22:48:39.579938 29190 sys_catalog.cc:455] T 00000000000000000000000000000000 P e452e50fa17e4ebb948423eb256aeeb5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e452e50fa17e4ebb948423eb256aeeb5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e452e50fa17e4ebb948423eb256aeeb5" member_type: VOTER last_known_addr { host: "127.10.198.62" port: 38513 } } }
I20260925 22:48:39.580044 29190 sys_catalog.cc:458] T 00000000000000000000000000000000 P e452e50fa17e4ebb948423eb256aeeb5 [sys.catalog]: This master's current role is: LEADER
I20260925 22:48:39.580514 29195 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 22:48:39.583799 29195 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 22:48:39.590157 29195 catalog_manager.cc:1347] Generated new cluster ID: 67331cfbeccf4a52b6fa111b93497e49
I20260925 22:48:39.590276 29195 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 22:48:39.604372 29195 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 22:48:39.605999 29195 catalog_manager.cc:1504] Loading token signing keys...
I20260925 22:48:39.618997 29195 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e452e50fa17e4ebb948423eb256aeeb5: Generated new TSK 0
I20260925 22:48:39.620146 29195 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.10.198.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.198.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.198.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790376259650642-11032-0/krb5kdc/kudu_127.10.198.1.keytab.
Entry for principal kudu/127.10.198.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790376259650642-11032-0/krb5kdc/kudu_127.10.198.1.keytab.
I20260925 22:48:39.656068 11032 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrWXm8X/build/debug/bin/kudu
/tmp/dist-test-taskrWXm8X/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790376259650642-11032-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790376259650642-11032-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790376259650642-11032-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790376259650642-11032-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.198.1:0
--local_ip_for_outbound_sockets=127.10.198.1
--webserver_interface=127.10.198.1
--webserver_port=0
--tserver_master_addrs=127.10.198.62:38513
--keytab_file=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790376259650642-11032-0/krb5kdc/kudu_127.10.198.1.keytab
--principal=kudu/127.10.198.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.198.20:44873
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790376259650642-11032-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790376259650642-11032-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790376259650642-11032-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:48:39.779953 29210 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:48:39.780311 29210 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:48:39.780400 29210 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:48:39.784929 29210 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:48:39.785053 29210 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:48:39.785158 29210 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.198.1
I20260925 22:48:39.791052 29210 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.198.20:44873
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790376259650642-11032-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790376259650642-11032-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790376259650642-11032-0/krb5kdc/kudu_127.10.198.1.keytab
--principal=kudu/127.10.198.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.198.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790376259650642-11032-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.198.1
--webserver_port=0
--tserver_master_addrs=127.10.198.62:38513
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29210
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.198.1
--log_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790376259650642-11032-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:09:16 UTC on ccycloud.quasar-afmhyw.root.comops.site
build id 4673379
I20260925 22:48:39.792419 29210 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:48:39.793570 29210 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:48:39.804266 29215 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:48:39.804225 29216 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:48:39.804225 29218 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:48:39.805404 29210 server_base.cc:1034] running on GCE node
I20260925 22:48:39.805927 29210 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:48:39.806784 29210 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:48:39.807955 29210 hybrid_clock.cc:648] HybridClock initialized: now 1790376519807928 us; error 42 us; skew 500 ppm
Sep 25 22:48:39 dist-test-slave-5vdr krb5kdc[29103](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376519, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.198.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:48:39.814692 29210 init.cc:377] Logged in from keytab as kudu/127.10.198.1@KRBTEST.COM (short username kudu)
I20260925 22:48:39.816011 29210 webserver.cc:460] Webserver started at http://127.10.198.1:46251/ using document root <none> and password file <none>
I20260925 22:48:39.816699 29210 fs_manager.cc:359] Metadata directory not provided
I20260925 22:48:39.816782 29210 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:48:39.817024 29210 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:48:39.819348 29210 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790376259650642-11032-0/minicluster-data/ts-0/data/instance:
uuid: "cd7662f07e8e467998876667b779e462"
format_stamp: "Formatted at 2026-09-25 22:48:39 on dist-test-slave-5vdr"
server_key: "eed283b85c92b7f2da5e8a252ffa0fa8"
server_key_iv: "f8d74ec814fe56a27a2578220dd9c5d2"
server_key_version: "encryptionkey@0"
I20260925 22:48:39.820086 29210 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790376259650642-11032-0/minicluster-data/ts-0/wal/instance:
uuid: "cd7662f07e8e467998876667b779e462"
format_stamp: "Formatted at 2026-09-25 22:48:39 on dist-test-slave-5vdr"
server_key: "eed283b85c92b7f2da5e8a252ffa0fa8"
server_key_iv: "f8d74ec814fe56a27a2578220dd9c5d2"
server_key_version: "encryptionkey@0"
I20260925 22:48:39.824481 29210 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.003s
I20260925 22:48:39.827625 29225 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:48:39.829157 29210 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 22:48:39.829356 29210 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790376259650642-11032-0/minicluster-data/ts-0/data,/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790376259650642-11032-0/minicluster-data/ts-0/wal
uuid: "cd7662f07e8e467998876667b779e462"
format_stamp: "Formatted at 2026-09-25 22:48:39 on dist-test-slave-5vdr"
server_key: "eed283b85c92b7f2da5e8a252ffa0fa8"
server_key_iv: "f8d74ec814fe56a27a2578220dd9c5d2"
server_key_version: "encryptionkey@0"
I20260925 22:48:39.829506 29210 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790376259650642-11032-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790376259650642-11032-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790376259650642-11032-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:48:39.855485 29210 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:48:39.858852 29210 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:48:39.859097 29210 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:48:39.859763 29210 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 22:48:39.860750 29210 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:48:39.860834 29210 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:48:39.860915 29210 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:48:39.860968 29210 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 22:48:39.871645 29210 rpc_server.cc:307] RPC server started. Bound to: 127.10.198.1:44565
I20260925 22:48:39.871675 29338 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.198.1:44565 every 8 connection(s)
I20260925 22:48:39.872763 29210 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790376259650642-11032-0/minicluster-data/ts-0/data/info.pb
I20260925 22:48:39.874684 11032 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrWXm8X/build/debug/bin/kudu as pid 29210
I20260925 22:48:39.874867 11032 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790376259650642-11032-0/minicluster-data/ts-0/wal/instance
I20260925 22:48:39.875213 11032 external_mini_cluster.cc:1428] Setting key c4f8a99276b89dd8f074a00f05d02582
Sep 25 22:48:39 dist-test-slave-5vdr krb5kdc[29103](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376519, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.198.1@KRBTEST.COM for kudu/127.10.198.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.10.198.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.198.2@KRBTEST.COM" created.
I20260925 22:48:39.890277 29187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:48:39.874954 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.198.1:56939 (local address 127.10.198.62:38513)
0925 22:48:39.875141 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0925 22:48:39.875147 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 22:48:39.876582 (+ 1435us) server_negotiation.cc:407] Connection header received
0925 22:48:39.877964 (+ 1382us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:48:39.877967 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:48:39.878022 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:48:39.878103 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:48:39.879928 (+ 1825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:48:39.880494 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:48:39.881927 (+ 1433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:48:39.882182 (+ 255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:48:39.882322 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:48:39.885300 (+ 2978us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:48:39.885304 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:48:39.885306 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:48:39.885336 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:48:39.887160 (+ 1824us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:48:39.888095 (+ 935us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:48:39.888099 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:48:39.888100 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:48:39.888203 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:48:39.888671 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:48:39.888678 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:48:39.888683 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:48:39.888982 (+ 299us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:48:39.889116 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0925 22:48:39.889971 (+ 855us) server_negotiation.cc:299] Negotiation successful
0925 22:48:39.890142 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260925 22:48:39.891441 29341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:48:39.875455 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.198.62:38513 (local address 127.10.198.1:56939)
0925 22:48:39.876407 (+ 952us) negotiation.cc:107] Waiting for socket to connect
0925 22:48:39.876461 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0925 22:48:39.877658 (+ 1197us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:48:39.878357 (+ 699us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:48:39.878369 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:48:39.879099 (+ 730us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:48:39.879690 (+ 591us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:48:39.879706 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:48:39.880657 (+ 951us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:48:39.880661 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:48:39.881770 (+ 1109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:48:39.881783 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:48:39.882382 (+ 599us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:48:39.882386 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:48:39.882578 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:48:39.883274 (+ 696us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:48:39.883294 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:48:39.885054 (+ 1760us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:48:39.887351 (+ 2297us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:48:39.887367 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:48:39.887384 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:48:39.887950 (+ 566us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:48:39.888318 (+ 368us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:48:39.888321 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:48:39.888323 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:48:39.888510 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:48:39.889155 (+ 645us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:48:39.889165 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:48:39.889614 (+ 449us) client_negotiation.cc:769] Sending connection context
0925 22:48:39.889887 (+ 273us) client_negotiation.cc:240] Negotiation successful
0925 22:48:39.890363 (+ 476us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":728,"thread_start_us":196,"threads_started":1}
I20260925 22:48:39.893080 29339 heartbeater.cc:347] Connected to a master server at 127.10.198.62:38513
I20260925 22:48:39.893399 29339 heartbeater.cc:464] Registering TS with master...
I20260925 22:48:39.893988 29339 heartbeater.cc:511] Master 127.10.198.62:38513 requested a full tablet report, sending...
I20260925 22:48:39.895993 29148 ts_manager.cc:194] Registered new tserver with Master: cd7662f07e8e467998876667b779e462 (127.10.198.1:44565)
I20260925 22:48:39.897811 29148 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.198.1@KRBTEST.COM'} at 127.10.198.1:56939
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.198.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790376259650642-11032-0/krb5kdc/kudu_127.10.198.2.keytab.
Entry for principal kudu/127.10.198.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790376259650642-11032-0/krb5kdc/kudu_127.10.198.2.keytab.
I20260925 22:48:39.906685 11032 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrWXm8X/build/debug/bin/kudu
/tmp/dist-test-taskrWXm8X/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790376259650642-11032-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790376259650642-11032-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790376259650642-11032-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790376259650642-11032-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.198.2:0
--local_ip_for_outbound_sockets=127.10.198.2
--webserver_interface=127.10.198.2
--webserver_port=0
--tserver_master_addrs=127.10.198.62:38513
--keytab_file=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790376259650642-11032-0/krb5kdc/kudu_127.10.198.2.keytab
--principal=kudu/127.10.198.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.198.20:44873
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790376259650642-11032-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790376259650642-11032-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790376259650642-11032-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:48:40.025681 29344 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:48:40.026054 29344 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:48:40.026152 29344 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:48:40.030642 29344 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:48:40.030803 29344 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:48:40.030910 29344 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.198.2
I20260925 22:48:40.037178 29344 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.198.20:44873
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790376259650642-11032-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790376259650642-11032-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790376259650642-11032-0/krb5kdc/kudu_127.10.198.2.keytab
--principal=kudu/127.10.198.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.198.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790376259650642-11032-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.198.2
--webserver_port=0
--tserver_master_addrs=127.10.198.62:38513
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29344
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.198.2
--log_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790376259650642-11032-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:09:16 UTC on ccycloud.quasar-afmhyw.root.comops.site
build id 4673379
I20260925 22:48:40.038436 29344 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:48:40.039562 29344 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:48:40.050254 29349 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:48:40.050433 29344 server_base.cc:1034] running on GCE node
W20260925 22:48:40.050240 29352 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:48:40.050278 29350 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:48:40.051151 29344 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:48:40.051932 29344 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:48:40.053164 29344 hybrid_clock.cc:648] HybridClock initialized: now 1790376520053145 us; error 38 us; skew 500 ppm
Sep 25 22:48:40 dist-test-slave-5vdr krb5kdc[29103](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376520, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.198.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:48:40.056334 29344 init.cc:377] Logged in from keytab as kudu/127.10.198.2@KRBTEST.COM (short username kudu)
I20260925 22:48:40.057476 29344 webserver.cc:460] Webserver started at http://127.10.198.2:33315/ using document root <none> and password file <none>
I20260925 22:48:40.058195 29344 fs_manager.cc:359] Metadata directory not provided
I20260925 22:48:40.058308 29344 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:48:40.058547 29344 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:48:40.060925 29344 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790376259650642-11032-0/minicluster-data/ts-1/data/instance:
uuid: "7c9c6850bc254babbd437f31fed53ff0"
format_stamp: "Formatted at 2026-09-25 22:48:40 on dist-test-slave-5vdr"
server_key: "d7c4b9515c1bc4dfcb3898e8de4951cb"
server_key_iv: "7a621ede9d4a2b611f12b4383041f76e"
server_key_version: "encryptionkey@0"
I20260925 22:48:40.061586 29344 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790376259650642-11032-0/minicluster-data/ts-1/wal/instance:
uuid: "7c9c6850bc254babbd437f31fed53ff0"
format_stamp: "Formatted at 2026-09-25 22:48:40 on dist-test-slave-5vdr"
server_key: "d7c4b9515c1bc4dfcb3898e8de4951cb"
server_key_iv: "7a621ede9d4a2b611f12b4383041f76e"
server_key_version: "encryptionkey@0"
I20260925 22:48:40.065989 29344 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260925 22:48:40.068979 29359 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:48:40.070501 29344 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 22:48:40.070729 29344 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790376259650642-11032-0/minicluster-data/ts-1/data,/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790376259650642-11032-0/minicluster-data/ts-1/wal
uuid: "7c9c6850bc254babbd437f31fed53ff0"
format_stamp: "Formatted at 2026-09-25 22:48:40 on dist-test-slave-5vdr"
server_key: "d7c4b9515c1bc4dfcb3898e8de4951cb"
server_key_iv: "7a621ede9d4a2b611f12b4383041f76e"
server_key_version: "encryptionkey@0"
I20260925 22:48:40.070883 29344 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790376259650642-11032-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790376259650642-11032-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790376259650642-11032-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:48:40.084116 29344 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:48:40.087707 29344 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:48:40.087994 29344 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:48:40.088678 29344 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 22:48:40.089582 29344 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:48:40.089666 29344 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:48:40.089740 29344 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:48:40.089793 29344 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 22:48:40.099424 29344 rpc_server.cc:307] RPC server started. Bound to: 127.10.198.2:46119
I20260925 22:48:40.099442 29472 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.198.2:46119 every 8 connection(s)
I20260925 22:48:40.100533 29344 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790376259650642-11032-0/minicluster-data/ts-1/data/info.pb
I20260925 22:48:40.103789 11032 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrWXm8X/build/debug/bin/kudu as pid 29344
I20260925 22:48:40.104051 11032 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790376259650642-11032-0/minicluster-data/ts-1/wal/instance
I20260925 22:48:40.104374 11032 external_mini_cluster.cc:1428] Setting key fdee937b7631eef5e112b2c2f4637be1
Sep 25 22:48:40 dist-test-slave-5vdr krb5kdc[29103](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376520, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.198.2@KRBTEST.COM for kudu/127.10.198.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.10.198.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.198.3@KRBTEST.COM" created.
I20260925 22:48:40.117724 29187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:48:40.102498 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.198.2:55005 (local address 127.10.198.62:38513)
0925 22:48:40.102723 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0925 22:48:40.102729 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 22:48:40.103898 (+ 1169us) server_negotiation.cc:407] Connection header received
0925 22:48:40.105013 (+ 1115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:48:40.105017 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:48:40.105074 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:48:40.105175 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:48:40.106725 (+ 1550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:48:40.107604 (+ 879us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:48:40.109662 (+ 2058us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:48:40.109903 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:48:40.110004 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:48:40.113514 (+ 3510us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:48:40.113518 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:48:40.113520 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:48:40.113552 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:48:40.115259 (+ 1707us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:48:40.115957 (+ 698us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:48:40.115960 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:48:40.115961 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:48:40.116047 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:48:40.116464 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:48:40.116470 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:48:40.116476 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:48:40.116693 (+ 217us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:48:40.116809 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0925 22:48:40.117397 (+ 588us) server_negotiation.cc:299] Negotiation successful
0925 22:48:40.117551 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260925 22:48:40.118786 29475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:48:40.102951 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.198.62:38513 (local address 127.10.198.2:55005)
0925 22:48:40.103683 (+ 732us) negotiation.cc:107] Waiting for socket to connect
0925 22:48:40.103732 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0925 22:48:40.104759 (+ 1027us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:48:40.105405 (+ 646us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:48:40.105414 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:48:40.105999 (+ 585us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:48:40.106548 (+ 549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:48:40.106563 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:48:40.107774 (+ 1211us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:48:40.107779 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:48:40.109486 (+ 1707us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:48:40.109506 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:48:40.110098 (+ 592us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:48:40.110105 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:48:40.110383 (+ 278us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:48:40.111156 (+ 773us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:48:40.111187 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:48:40.113219 (+ 2032us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:48:40.115420 (+ 2201us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:48:40.115429 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:48:40.115451 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:48:40.115782 (+ 331us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:48:40.116155 (+ 373us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:48:40.116160 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:48:40.116162 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:48:40.116346 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:48:40.116811 (+ 465us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:48:40.116822 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:48:40.117123 (+ 301us) client_negotiation.cc:769] Sending connection context
0925 22:48:40.117330 (+ 207us) client_negotiation.cc:240] Negotiation successful
0925 22:48:40.117721 (+ 391us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":524,"thread_start_us":146,"threads_started":1}
I20260925 22:48:40.120214 29473 heartbeater.cc:347] Connected to a master server at 127.10.198.62:38513
I20260925 22:48:40.120563 29473 heartbeater.cc:464] Registering TS with master...
I20260925 22:48:40.121307 29473 heartbeater.cc:511] Master 127.10.198.62:38513 requested a full tablet report, sending...
I20260925 22:48:40.122762 29148 ts_manager.cc:194] Registered new tserver with Master: 7c9c6850bc254babbd437f31fed53ff0 (127.10.198.2:46119)
I20260925 22:48:40.123745 29148 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.198.2@KRBTEST.COM'} at 127.10.198.2:55005
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.198.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790376259650642-11032-0/krb5kdc/kudu_127.10.198.3.keytab.
Entry for principal kudu/127.10.198.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790376259650642-11032-0/krb5kdc/kudu_127.10.198.3.keytab.
I20260925 22:48:40.135541 11032 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrWXm8X/build/debug/bin/kudu
/tmp/dist-test-taskrWXm8X/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790376259650642-11032-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790376259650642-11032-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790376259650642-11032-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790376259650642-11032-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.198.3:0
--local_ip_for_outbound_sockets=127.10.198.3
--webserver_interface=127.10.198.3
--webserver_port=0
--tserver_master_addrs=127.10.198.62:38513
--keytab_file=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790376259650642-11032-0/krb5kdc/kudu_127.10.198.3.keytab
--principal=kudu/127.10.198.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.198.20:44873
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790376259650642-11032-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790376259650642-11032-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790376259650642-11032-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:48:40.256138 29478 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:48:40.256489 29478 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:48:40.256574 29478 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:48:40.261137 29478 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:48:40.261262 29478 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:48:40.261368 29478 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.198.3
I20260925 22:48:40.267280 29478 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.198.20:44873
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790376259650642-11032-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790376259650642-11032-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790376259650642-11032-0/krb5kdc/kudu_127.10.198.3.keytab
--principal=kudu/127.10.198.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.198.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790376259650642-11032-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.198.3
--webserver_port=0
--tserver_master_addrs=127.10.198.62:38513
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29478
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.198.3
--log_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790376259650642-11032-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:09:16 UTC on ccycloud.quasar-afmhyw.root.comops.site
build id 4673379
I20260925 22:48:40.268551 29478 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:48:40.269673 29478 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:48:40.280652 29483 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:48:40.280836 29478 server_base.cc:1034] running on GCE node
W20260925 22:48:40.280652 29484 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:48:40.280618 29486 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:48:40.281706 29478 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:48:40.282437 29478 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:48:40.283624 29478 hybrid_clock.cc:648] HybridClock initialized: now 1790376520283593 us; error 53 us; skew 500 ppm
Sep 25 22:48:40 dist-test-slave-5vdr krb5kdc[29103](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376520, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.198.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:48:40.286875 29478 init.cc:377] Logged in from keytab as kudu/127.10.198.3@KRBTEST.COM (short username kudu)
I20260925 22:48:40.288023 29478 webserver.cc:460] Webserver started at http://127.10.198.3:44883/ using document root <none> and password file <none>
I20260925 22:48:40.288661 29478 fs_manager.cc:359] Metadata directory not provided
I20260925 22:48:40.288733 29478 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:48:40.288980 29478 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:48:40.291296 29478 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790376259650642-11032-0/minicluster-data/ts-2/data/instance:
uuid: "4733da8ac5794208a97cb3a2c063779c"
format_stamp: "Formatted at 2026-09-25 22:48:40 on dist-test-slave-5vdr"
server_key: "bacaecdc30dd93ae02c4b08a8928fc5a"
server_key_iv: "8ba27a80d21ddfdf34c20b4babaadd2a"
server_key_version: "encryptionkey@0"
I20260925 22:48:40.292054 29478 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790376259650642-11032-0/minicluster-data/ts-2/wal/instance:
uuid: "4733da8ac5794208a97cb3a2c063779c"
format_stamp: "Formatted at 2026-09-25 22:48:40 on dist-test-slave-5vdr"
server_key: "bacaecdc30dd93ae02c4b08a8928fc5a"
server_key_iv: "8ba27a80d21ddfdf34c20b4babaadd2a"
server_key_version: "encryptionkey@0"
I20260925 22:48:40.296681 29478 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260925 22:48:40.299526 29493 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:48:40.301057 29478 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 22:48:40.301241 29478 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790376259650642-11032-0/minicluster-data/ts-2/data,/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790376259650642-11032-0/minicluster-data/ts-2/wal
uuid: "4733da8ac5794208a97cb3a2c063779c"
format_stamp: "Formatted at 2026-09-25 22:48:40 on dist-test-slave-5vdr"
server_key: "bacaecdc30dd93ae02c4b08a8928fc5a"
server_key_iv: "8ba27a80d21ddfdf34c20b4babaadd2a"
server_key_version: "encryptionkey@0"
I20260925 22:48:40.301390 29478 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790376259650642-11032-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790376259650642-11032-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790376259650642-11032-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:48:40.321707 29478 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:48:40.325222 29478 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:48:40.325467 29478 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:48:40.326128 29478 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 22:48:40.327148 29478 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:48:40.327224 29478 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:48:40.327298 29478 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:48:40.327348 29478 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 22:48:40.337512 29478 rpc_server.cc:307] RPC server started. Bound to: 127.10.198.3:39625
I20260925 22:48:40.337534 29606 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.198.3:39625 every 8 connection(s)
I20260925 22:48:40.338554 29478 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790376259650642-11032-0/minicluster-data/ts-2/data/info.pb
I20260925 22:48:40.343222 11032 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrWXm8X/build/debug/bin/kudu as pid 29478
I20260925 22:48:40.343391 11032 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790376259650642-11032-0/minicluster-data/ts-2/wal/instance
I20260925 22:48:40.343746 11032 external_mini_cluster.cc:1428] Setting key 90e0c6f61af7b98428ee9aa0a302d670
Sep 25 22:48:40 dist-test-slave-5vdr krb5kdc[29103](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376520, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.198.3@KRBTEST.COM for kudu/127.10.198.62@KRBTEST.COM
I20260925 22:48:40.354075 29187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:48:40.340617 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.198.3:59593 (local address 127.10.198.62:38513)
0925 22:48:40.340812 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0925 22:48:40.340817 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:48:40.342036 (+ 1219us) server_negotiation.cc:407] Connection header received
0925 22:48:40.342968 (+ 932us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:48:40.342971 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:48:40.343024 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:48:40.343103 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:48:40.344784 (+ 1681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:48:40.345436 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:48:40.346932 (+ 1496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:48:40.347159 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:48:40.347255 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:48:40.349969 (+ 2714us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:48:40.349973 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:48:40.349975 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:48:40.350004 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:48:40.351632 (+ 1628us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:48:40.352329 (+ 697us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:48:40.352335 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:48:40.352337 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:48:40.352429 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:48:40.352836 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:48:40.352840 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:48:40.352841 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:48:40.353052 (+ 211us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:48:40.353168 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0925 22:48:40.353757 (+ 589us) server_negotiation.cc:299] Negotiation successful
0925 22:48:40.353902 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260925 22:48:40.354995 29609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:48:40.341066 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.198.62:38513 (local address 127.10.198.3:59593)
0925 22:48:40.341803 (+ 737us) negotiation.cc:107] Waiting for socket to connect
0925 22:48:40.341884 (+ 81us) client_negotiation.cc:174] Beginning negotiation
0925 22:48:40.342723 (+ 839us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:48:40.343311 (+ 588us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:48:40.343322 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:48:40.343961 (+ 639us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:48:40.344608 (+ 647us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:48:40.344628 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:48:40.345580 (+ 952us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:48:40.345584 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:48:40.346784 (+ 1200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:48:40.346797 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:48:40.347319 (+ 522us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:48:40.347322 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:48:40.347478 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:48:40.348105 (+ 627us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:48:40.348123 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:48:40.349745 (+ 1622us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:48:40.351779 (+ 2034us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:48:40.351786 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:48:40.351807 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:48:40.352199 (+ 392us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:48:40.352531 (+ 332us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:48:40.352535 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:48:40.352537 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:48:40.352720 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:48:40.353170 (+ 450us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:48:40.353180 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:48:40.353492 (+ 312us) client_negotiation.cc:769] Sending connection context
0925 22:48:40.353698 (+ 206us) client_negotiation.cc:240] Negotiation successful
0925 22:48:40.354085 (+ 387us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":516,"thread_start_us":129,"threads_started":1}
I20260925 22:48:40.356375 29607 heartbeater.cc:347] Connected to a master server at 127.10.198.62:38513
I20260925 22:48:40.356788 29607 heartbeater.cc:464] Registering TS with master...
I20260925 22:48:40.357494 29607 heartbeater.cc:511] Master 127.10.198.62:38513 requested a full tablet report, sending...
I20260925 22:48:40.358794 29148 ts_manager.cc:194] Registered new tserver with Master: 4733da8ac5794208a97cb3a2c063779c (127.10.198.3:39625)
I20260925 22:48:40.359388 29148 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.198.3@KRBTEST.COM'} at 127.10.198.3:59593
I20260925 22:48:40.370594 11032 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 22:48:40.382124 29187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:48:40.372837 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36312 (local address 127.10.198.62:38513)
0925 22:48:40.372975 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0925 22:48:40.372984 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 22:48:40.373084 (+ 100us) server_negotiation.cc:407] Connection header received
0925 22:48:40.373196 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:48:40.373198 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:48:40.373245 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:48:40.373328 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:48:40.374074 (+ 746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:48:40.374659 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:48:40.375672 (+ 1013us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:48:40.376021 (+ 349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:48:40.376137 (+ 116us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:48:40.378103 (+ 1966us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:48:40.378107 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:48:40.378109 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:48:40.378142 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:48:40.379931 (+ 1789us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:48:40.380786 (+ 855us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:48:40.380790 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:48:40.380791 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:48:40.380892 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:48:40.381248 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:48:40.381251 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:48:40.381252 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:48:40.381494 (+ 242us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:48:40.381621 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0925 22:48:40.381899 (+ 278us) server_negotiation.cc:299] Negotiation successful
0925 22:48:40.381994 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260925 22:48:40.385787 29148 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36312:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 22:48:40.388995 29148 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 22:48:40.406104 29619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:48:40.399048 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.198.3:39625 (local address 127.0.0.1:52826)
0925 22:48:40.399664 (+ 616us) negotiation.cc:107] Waiting for socket to connect
0925 22:48:40.399725 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0925 22:48:40.399900 (+ 175us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:48:40.401082 (+ 1182us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:48:40.401089 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:48:40.401125 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:48:40.401400 (+ 275us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:48:40.401408 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:48:40.403006 (+ 1598us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:48:40.403010 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:48:40.404196 (+ 1186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:48:40.404206 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:48:40.405339 (+ 1133us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:48:40.405345 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:48:40.405509 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:48:40.405623 (+ 114us) client_negotiation.cc:769] Sending connection context
0925 22:48:40.405779 (+ 156us) client_negotiation.cc:240] Negotiation successful
0925 22:48:40.405938 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":421,"mutex_wait_us":159,"thread_start_us":123,"threads_started":1}
I20260925 22:48:40.406117 29622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:48:40.399834 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52826 (local address 127.10.198.3:39625)
0925 22:48:40.400632 (+ 798us) server_negotiation.cc:206] Beginning negotiation
0925 22:48:40.400649 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0925 22:48:40.400669 (+ 20us) server_negotiation.cc:407] Connection header received
0925 22:48:40.400747 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:48:40.400752 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:48:40.400908 (+ 156us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:48:40.401548 (+ 640us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:48:40.401723 (+ 175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:48:40.402812 (+ 1089us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:48:40.404350 (+ 1538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:48:40.405152 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:48:40.405273 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:48:40.405442 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0925 22:48:40.405822 (+ 380us) server_negotiation.cc:299] Negotiation successful
0925 22:48:40.405941 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":673,"thread_start_us":135,"threads_started":1}
I20260925 22:48:40.406491 29620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:48:40.399692 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41422 (local address 127.10.198.1:44565)
0925 22:48:40.400442 (+ 750us) server_negotiation.cc:206] Beginning negotiation
0925 22:48:40.400451 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 22:48:40.400479 (+ 28us) server_negotiation.cc:407] Connection header received
0925 22:48:40.400568 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:48:40.400573 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:48:40.400810 (+ 237us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:48:40.400977 (+ 167us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:48:40.401538 (+ 561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:48:40.402777 (+ 1239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:48:40.404452 (+ 1675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:48:40.405314 (+ 862us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:48:40.405424 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:48:40.405580 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0925 22:48:40.405932 (+ 352us) server_negotiation.cc:299] Negotiation successful
0925 22:48:40.406045 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":638,"thread_start_us":141,"threads_started":1}
I20260925 22:48:40.406548 29618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:48:40.399463 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.198.1:44565 (local address 127.0.0.1:41422)
0925 22:48:40.400282 (+ 819us) negotiation.cc:107] Waiting for socket to connect
0925 22:48:40.400298 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 22:48:40.400388 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:48:40.401145 (+ 757us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:48:40.401147 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:48:40.401159 (+ 12us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:48:40.401400 (+ 241us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:48:40.401408 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:48:40.402914 (+ 1506us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:48:40.402917 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:48:40.404280 (+ 1363us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:48:40.404290 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:48:40.405530 (+ 1240us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:48:40.405533 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:48:40.405658 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:48:40.405683 (+ 25us) client_negotiation.cc:769] Sending connection context
0925 22:48:40.405770 (+ 87us) client_negotiation.cc:240] Negotiation successful
0925 22:48:40.406343 (+ 573us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":756,"mutex_wait_us":1102,"thread_start_us":98,"threads_started":1}
I20260925 22:48:40.407259 29623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:48:40.399239 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.198.2:46119 (local address 127.0.0.1:39668)
0925 22:48:40.400674 (+ 1435us) negotiation.cc:107] Waiting for socket to connect
0925 22:48:40.400687 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 22:48:40.400784 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:48:40.401508 (+ 724us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:48:40.401513 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:48:40.401532 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:48:40.401619 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:48:40.401624 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:48:40.402696 (+ 1072us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:48:40.402700 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:48:40.404896 (+ 2196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:48:40.404908 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:48:40.406839 (+ 1931us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:48:40.406845 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:48:40.406940 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:48:40.406965 (+ 25us) client_negotiation.cc:769] Sending connection context
0925 22:48:40.407050 (+ 85us) client_negotiation.cc:240] Negotiation successful
0925 22:48:40.407127 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":960,"mutex_wait_us":65,"thread_start_us":141,"threads_started":1}
I20260925 22:48:40.407346 29621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:48:40.399818 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39668 (local address 127.10.198.2:46119)
0925 22:48:40.400442 (+ 624us) server_negotiation.cc:206] Beginning negotiation
0925 22:48:40.400453 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 22:48:40.400884 (+ 431us) server_negotiation.cc:407] Connection header received
0925 22:48:40.400973 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:48:40.400979 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:48:40.401189 (+ 210us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:48:40.401355 (+ 166us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:48:40.401746 (+ 391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:48:40.402520 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:48:40.405046 (+ 2526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:48:40.406106 (+ 1060us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:48:40.406799 (+ 693us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:48:40.406964 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0925 22:48:40.407093 (+ 129us) server_negotiation.cc:299] Negotiation successful
0925 22:48:40.407203 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":509,"thread_start_us":95,"threads_started":1}
I20260925 22:48:40.409659 29541 tablet_service.cc:1467] Processing CreateTablet for tablet be3c13113bcd4f36824fcc41551805e8 (DEFAULT_TABLE table=test-table [id=df8d84e430654bda90b062e817e7e53e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 22:48:40.409658 29273 tablet_service.cc:1467] Processing CreateTablet for tablet be3c13113bcd4f36824fcc41551805e8 (DEFAULT_TABLE table=test-table [id=df8d84e430654bda90b062e817e7e53e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 22:48:40.409940 29407 tablet_service.cc:1467] Processing CreateTablet for tablet be3c13113bcd4f36824fcc41551805e8 (DEFAULT_TABLE table=test-table [id=df8d84e430654bda90b062e817e7e53e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 22:48:40.410838 29407 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet be3c13113bcd4f36824fcc41551805e8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:48:40.410859 29273 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet be3c13113bcd4f36824fcc41551805e8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:48:40.410866 29541 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet be3c13113bcd4f36824fcc41551805e8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:48:40.418576 29625 tablet_bootstrap.cc:492] T be3c13113bcd4f36824fcc41551805e8 P cd7662f07e8e467998876667b779e462: Bootstrap starting.
I20260925 22:48:40.419101 29624 tablet_bootstrap.cc:492] T be3c13113bcd4f36824fcc41551805e8 P 7c9c6850bc254babbd437f31fed53ff0: Bootstrap starting.
I20260925 22:48:40.419685 29626 tablet_bootstrap.cc:492] T be3c13113bcd4f36824fcc41551805e8 P 4733da8ac5794208a97cb3a2c063779c: Bootstrap starting.
I20260925 22:48:40.421887 29624 tablet_bootstrap.cc:654] T be3c13113bcd4f36824fcc41551805e8 P 7c9c6850bc254babbd437f31fed53ff0: Neither blocks nor log segments found. Creating new log.
I20260925 22:48:40.421959 29625 tablet_bootstrap.cc:654] T be3c13113bcd4f36824fcc41551805e8 P cd7662f07e8e467998876667b779e462: Neither blocks nor log segments found. Creating new log.
I20260925 22:48:40.421959 29626 tablet_bootstrap.cc:654] T be3c13113bcd4f36824fcc41551805e8 P 4733da8ac5794208a97cb3a2c063779c: Neither blocks nor log segments found. Creating new log.
I20260925 22:48:40.422816 29624 log.cc:824] T be3c13113bcd4f36824fcc41551805e8 P 7c9c6850bc254babbd437f31fed53ff0: Log is configured to *not* fsync() on all Append() calls
I20260925 22:48:40.422926 29625 log.cc:824] T be3c13113bcd4f36824fcc41551805e8 P cd7662f07e8e467998876667b779e462: Log is configured to *not* fsync() on all Append() calls
I20260925 22:48:40.423488 29626 log.cc:824] T be3c13113bcd4f36824fcc41551805e8 P 4733da8ac5794208a97cb3a2c063779c: Log is configured to *not* fsync() on all Append() calls
I20260925 22:48:40.425328 29624 tablet_bootstrap.cc:492] T be3c13113bcd4f36824fcc41551805e8 P 7c9c6850bc254babbd437f31fed53ff0: No bootstrap required, opened a new log
I20260925 22:48:40.425390 29625 tablet_bootstrap.cc:492] T be3c13113bcd4f36824fcc41551805e8 P cd7662f07e8e467998876667b779e462: No bootstrap required, opened a new log
I20260925 22:48:40.425462 29626 tablet_bootstrap.cc:492] T be3c13113bcd4f36824fcc41551805e8 P 4733da8ac5794208a97cb3a2c063779c: No bootstrap required, opened a new log
I20260925 22:48:40.425596 29624 ts_tablet_manager.cc:1397] T be3c13113bcd4f36824fcc41551805e8 P 7c9c6850bc254babbd437f31fed53ff0: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260925 22:48:40.425596 29625 ts_tablet_manager.cc:1397] T be3c13113bcd4f36824fcc41551805e8 P cd7662f07e8e467998876667b779e462: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260925 22:48:40.425616 29626 ts_tablet_manager.cc:1397] T be3c13113bcd4f36824fcc41551805e8 P 4733da8ac5794208a97cb3a2c063779c: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260925 22:48:40.428949 29624 raft_consensus.cc:348] T be3c13113bcd4f36824fcc41551805e8 P 7c9c6850bc254babbd437f31fed53ff0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4733da8ac5794208a97cb3a2c063779c" member_type: VOTER last_known_addr { host: "127.10.198.3" port: 39625 } } peers { permanent_uuid: "7c9c6850bc254babbd437f31fed53ff0" member_type: VOTER last_known_addr { host: "127.10.198.2" port: 46119 } } peers { permanent_uuid: "cd7662f07e8e467998876667b779e462" member_type: VOTER last_known_addr { host: "127.10.198.1" port: 44565 } }
I20260925 22:48:40.429288 29624 raft_consensus.cc:374] T be3c13113bcd4f36824fcc41551805e8 P 7c9c6850bc254babbd437f31fed53ff0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:48:40.429360 29624 raft_consensus.cc:729] T be3c13113bcd4f36824fcc41551805e8 P 7c9c6850bc254babbd437f31fed53ff0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7c9c6850bc254babbd437f31fed53ff0, State: Initialized, Role: FOLLOWER
I20260925 22:48:40.429883 29624 consensus_queue.cc:260] T be3c13113bcd4f36824fcc41551805e8 P 7c9c6850bc254babbd437f31fed53ff0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4733da8ac5794208a97cb3a2c063779c" member_type: VOTER last_known_addr { host: "127.10.198.3" port: 39625 } } peers { permanent_uuid: "7c9c6850bc254babbd437f31fed53ff0" member_type: VOTER last_known_addr { host: "127.10.198.2" port: 46119 } } peers { permanent_uuid: "cd7662f07e8e467998876667b779e462" member_type: VOTER last_known_addr { host: "127.10.198.1" port: 44565 } }
I20260925 22:48:40.429927 29626 raft_consensus.cc:348] T be3c13113bcd4f36824fcc41551805e8 P 4733da8ac5794208a97cb3a2c063779c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4733da8ac5794208a97cb3a2c063779c" member_type: VOTER last_known_addr { host: "127.10.198.3" port: 39625 } } peers { permanent_uuid: "7c9c6850bc254babbd437f31fed53ff0" member_type: VOTER last_known_addr { host: "127.10.198.2" port: 46119 } } peers { permanent_uuid: "cd7662f07e8e467998876667b779e462" member_type: VOTER last_known_addr { host: "127.10.198.1" port: 44565 } }
I20260925 22:48:40.430286 29626 raft_consensus.cc:374] T be3c13113bcd4f36824fcc41551805e8 P 4733da8ac5794208a97cb3a2c063779c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:48:40.430361 29626 raft_consensus.cc:729] T be3c13113bcd4f36824fcc41551805e8 P 4733da8ac5794208a97cb3a2c063779c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4733da8ac5794208a97cb3a2c063779c, State: Initialized, Role: FOLLOWER
I20260925 22:48:40.430475 29625 raft_consensus.cc:348] T be3c13113bcd4f36824fcc41551805e8 P cd7662f07e8e467998876667b779e462 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4733da8ac5794208a97cb3a2c063779c" member_type: VOTER last_known_addr { host: "127.10.198.3" port: 39625 } } peers { permanent_uuid: "7c9c6850bc254babbd437f31fed53ff0" member_type: VOTER last_known_addr { host: "127.10.198.2" port: 46119 } } peers { permanent_uuid: "cd7662f07e8e467998876667b779e462" member_type: VOTER last_known_addr { host: "127.10.198.1" port: 44565 } }
I20260925 22:48:40.430743 29625 raft_consensus.cc:374] T be3c13113bcd4f36824fcc41551805e8 P cd7662f07e8e467998876667b779e462 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:48:40.430794 29625 raft_consensus.cc:729] T be3c13113bcd4f36824fcc41551805e8 P cd7662f07e8e467998876667b779e462 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd7662f07e8e467998876667b779e462, State: Initialized, Role: FOLLOWER
I20260925 22:48:40.430886 29626 consensus_queue.cc:260] T be3c13113bcd4f36824fcc41551805e8 P 4733da8ac5794208a97cb3a2c063779c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4733da8ac5794208a97cb3a2c063779c" member_type: VOTER last_known_addr { host: "127.10.198.3" port: 39625 } } peers { permanent_uuid: "7c9c6850bc254babbd437f31fed53ff0" member_type: VOTER last_known_addr { host: "127.10.198.2" port: 46119 } } peers { permanent_uuid: "cd7662f07e8e467998876667b779e462" member_type: VOTER last_known_addr { host: "127.10.198.1" port: 44565 } }
I20260925 22:48:40.431120 29624 ts_tablet_manager.cc:1428] T be3c13113bcd4f36824fcc41551805e8 P 7c9c6850bc254babbd437f31fed53ff0: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260925 22:48:40.431303 29625 consensus_queue.cc:260] T be3c13113bcd4f36824fcc41551805e8 P cd7662f07e8e467998876667b779e462 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4733da8ac5794208a97cb3a2c063779c" member_type: VOTER last_known_addr { host: "127.10.198.3" port: 39625 } } peers { permanent_uuid: "7c9c6850bc254babbd437f31fed53ff0" member_type: VOTER last_known_addr { host: "127.10.198.2" port: 46119 } } peers { permanent_uuid: "cd7662f07e8e467998876667b779e462" member_type: VOTER last_known_addr { host: "127.10.198.1" port: 44565 } }
I20260925 22:48:40.431962 29626 ts_tablet_manager.cc:1428] T be3c13113bcd4f36824fcc41551805e8 P 4733da8ac5794208a97cb3a2c063779c: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260925 22:48:40.432044 29339 heartbeater.cc:503] Master 127.10.198.62:38513 was elected leader, sending a full tablet report...
I20260925 22:48:40.432335 29607 heartbeater.cc:503] Master 127.10.198.62:38513 was elected leader, sending a full tablet report...
I20260925 22:48:40.432384 29473 heartbeater.cc:503] Master 127.10.198.62:38513 was elected leader, sending a full tablet report...
I20260925 22:48:40.432845 29625 ts_tablet_manager.cc:1428] T be3c13113bcd4f36824fcc41551805e8 P cd7662f07e8e467998876667b779e462: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260925 22:48:40.543131 29632 raft_consensus.cc:482] T be3c13113bcd4f36824fcc41551805e8 P cd7662f07e8e467998876667b779e462 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 22:48:40.543375 29632 raft_consensus.cc:504] T be3c13113bcd4f36824fcc41551805e8 P cd7662f07e8e467998876667b779e462 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4733da8ac5794208a97cb3a2c063779c" member_type: VOTER last_known_addr { host: "127.10.198.3" port: 39625 } } peers { permanent_uuid: "7c9c6850bc254babbd437f31fed53ff0" member_type: VOTER last_known_addr { host: "127.10.198.2" port: 46119 } } peers { permanent_uuid: "cd7662f07e8e467998876667b779e462" member_type: VOTER last_known_addr { host: "127.10.198.1" port: 44565 } }
I20260925 22:48:40.544605 29632 leader_election.cc:288] T be3c13113bcd4f36824fcc41551805e8 P cd7662f07e8e467998876667b779e462 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4733da8ac5794208a97cb3a2c063779c (127.10.198.3:39625), 7c9c6850bc254babbd437f31fed53ff0 (127.10.198.2:46119)
I20260925 22:48:40.549098 29621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:48:40.545092 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.198.1:50271 (local address 127.10.198.2:46119)
0925 22:48:40.545238 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0925 22:48:40.545243 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:48:40.545537 (+ 294us) server_negotiation.cc:407] Connection header received
0925 22:48:40.545667 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:48:40.545671 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:48:40.545727 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:48:40.545808 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:48:40.546151 (+ 343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:48:40.546725 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:48:40.548011 (+ 1286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:48:40.548449 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:48:40.548597 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:48:40.548657 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0925 22:48:40.548865 (+ 208us) server_negotiation.cc:299] Negotiation successful
0925 22:48:40.548946 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260925 22:48:40.549124 29622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:48:40.545180 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.198.1:58711 (local address 127.10.198.3:39625)
0925 22:48:40.545312 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0925 22:48:40.545316 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:48:40.545706 (+ 390us) server_negotiation.cc:407] Connection header received
0925 22:48:40.545851 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:48:40.545853 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:48:40.545905 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:48:40.545990 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:48:40.546299 (+ 309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:48:40.546914 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:48:40.548074 (+ 1160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:48:40.548513 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:48:40.548654 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:48:40.548710 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0925 22:48:40.548914 (+ 204us) server_negotiation.cc:299] Negotiation successful
0925 22:48:40.548992 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260925 22:48:40.549150 29634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:48:40.544971 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.198.2:46119 (local address 127.10.198.1:50271)
0925 22:48:40.545414 (+ 443us) negotiation.cc:107] Waiting for socket to connect
0925 22:48:40.545427 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 22:48:40.545549 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:48:40.545842 (+ 293us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:48:40.545845 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:48:40.545863 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:48:40.546022 (+ 159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:48:40.546030 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:48:40.546856 (+ 826us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:48:40.546862 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:48:40.547807 (+ 945us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:48:40.547820 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:48:40.548594 (+ 774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:48:40.548598 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:48:40.548722 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:48:40.548745 (+ 23us) client_negotiation.cc:769] Sending connection context
0925 22:48:40.548847 (+ 102us) client_negotiation.cc:240] Negotiation successful
0925 22:48:40.548931 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":369,"mutex_wait_us":34,"thread_start_us":131,"threads_started":1}
I20260925 22:48:40.549156 29633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:48:40.544971 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.198.3:39625 (local address 127.10.198.1:58711)
0925 22:48:40.545581 (+ 610us) negotiation.cc:107] Waiting for socket to connect
0925 22:48:40.545596 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 22:48:40.545719 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:48:40.546010 (+ 291us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:48:40.546013 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:48:40.546028 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:48:40.546160 (+ 132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:48:40.546167 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:48:40.547036 (+ 869us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:48:40.547041 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:48:40.547929 (+ 888us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:48:40.547939 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:48:40.548649 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:48:40.548652 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:48:40.548775 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:48:40.548794 (+ 19us) client_negotiation.cc:769] Sending connection context
0925 22:48:40.548895 (+ 101us) client_negotiation.cc:240] Negotiation successful
0925 22:48:40.548976 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":494,"mutex_wait_us":60,"thread_start_us":114,"threads_started":1}
I20260925 22:48:40.550107 29427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be3c13113bcd4f36824fcc41551805e8" candidate_uuid: "cd7662f07e8e467998876667b779e462" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7c9c6850bc254babbd437f31fed53ff0" is_pre_election: true
I20260925 22:48:40.550441 29561 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be3c13113bcd4f36824fcc41551805e8" candidate_uuid: "cd7662f07e8e467998876667b779e462" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4733da8ac5794208a97cb3a2c063779c" is_pre_election: true
I20260925 22:48:40.550477 29427 raft_consensus.cc:2454] T be3c13113bcd4f36824fcc41551805e8 P 7c9c6850bc254babbd437f31fed53ff0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cd7662f07e8e467998876667b779e462 in term 0.
I20260925 22:48:40.550721 29561 raft_consensus.cc:2454] T be3c13113bcd4f36824fcc41551805e8 P 4733da8ac5794208a97cb3a2c063779c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cd7662f07e8e467998876667b779e462 in term 0.
I20260925 22:48:40.551117 29228 leader_election.cc:302] T be3c13113bcd4f36824fcc41551805e8 P cd7662f07e8e467998876667b779e462 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7c9c6850bc254babbd437f31fed53ff0, cd7662f07e8e467998876667b779e462; no voters:
I20260925 22:48:40.551470 29632 raft_consensus.cc:2781] T be3c13113bcd4f36824fcc41551805e8 P cd7662f07e8e467998876667b779e462 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 22:48:40.551600 29632 raft_consensus.cc:482] T be3c13113bcd4f36824fcc41551805e8 P cd7662f07e8e467998876667b779e462 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 22:48:40.551648 29632 raft_consensus.cc:3037] T be3c13113bcd4f36824fcc41551805e8 P cd7662f07e8e467998876667b779e462 [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:48:40.552742 29632 raft_consensus.cc:504] T be3c13113bcd4f36824fcc41551805e8 P cd7662f07e8e467998876667b779e462 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4733da8ac5794208a97cb3a2c063779c" member_type: VOTER last_known_addr { host: "127.10.198.3" port: 39625 } } peers { permanent_uuid: "7c9c6850bc254babbd437f31fed53ff0" member_type: VOTER last_known_addr { host: "127.10.198.2" port: 46119 } } peers { permanent_uuid: "cd7662f07e8e467998876667b779e462" member_type: VOTER last_known_addr { host: "127.10.198.1" port: 44565 } }
I20260925 22:48:40.553141 29632 leader_election.cc:288] T be3c13113bcd4f36824fcc41551805e8 P cd7662f07e8e467998876667b779e462 [CANDIDATE]: Term 1 election: Requested vote from peers 4733da8ac5794208a97cb3a2c063779c (127.10.198.3:39625), 7c9c6850bc254babbd437f31fed53ff0 (127.10.198.2:46119)
I20260925 22:48:40.553577 29561 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be3c13113bcd4f36824fcc41551805e8" candidate_uuid: "cd7662f07e8e467998876667b779e462" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4733da8ac5794208a97cb3a2c063779c"
I20260925 22:48:40.553738 29561 raft_consensus.cc:3037] T be3c13113bcd4f36824fcc41551805e8 P 4733da8ac5794208a97cb3a2c063779c [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:48:40.555143 29561 raft_consensus.cc:2454] T be3c13113bcd4f36824fcc41551805e8 P 4733da8ac5794208a97cb3a2c063779c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cd7662f07e8e467998876667b779e462 in term 1.
I20260925 22:48:40.555546 29427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be3c13113bcd4f36824fcc41551805e8" candidate_uuid: "cd7662f07e8e467998876667b779e462" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7c9c6850bc254babbd437f31fed53ff0"
I20260925 22:48:40.555596 29229 leader_election.cc:302] T be3c13113bcd4f36824fcc41551805e8 P cd7662f07e8e467998876667b779e462 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4733da8ac5794208a97cb3a2c063779c, cd7662f07e8e467998876667b779e462; no voters:
I20260925 22:48:40.555716 29427 raft_consensus.cc:3037] T be3c13113bcd4f36824fcc41551805e8 P 7c9c6850bc254babbd437f31fed53ff0 [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:48:40.556892 29427 raft_consensus.cc:2454] T be3c13113bcd4f36824fcc41551805e8 P 7c9c6850bc254babbd437f31fed53ff0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cd7662f07e8e467998876667b779e462 in term 1.
I20260925 22:48:40.557219 29632 raft_consensus.cc:2781] T be3c13113bcd4f36824fcc41551805e8 P cd7662f07e8e467998876667b779e462 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 22:48:40.557535 29632 raft_consensus.cc:686] T be3c13113bcd4f36824fcc41551805e8 P cd7662f07e8e467998876667b779e462 [term 1 LEADER]: Becoming Leader. State: Replica: cd7662f07e8e467998876667b779e462, State: Running, Role: LEADER
I20260925 22:48:40.558118 29632 consensus_queue.cc:237] T be3c13113bcd4f36824fcc41551805e8 P cd7662f07e8e467998876667b779e462 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4733da8ac5794208a97cb3a2c063779c" member_type: VOTER last_known_addr { host: "127.10.198.3" port: 39625 } } peers { permanent_uuid: "7c9c6850bc254babbd437f31fed53ff0" member_type: VOTER last_known_addr { host: "127.10.198.2" port: 46119 } } peers { permanent_uuid: "cd7662f07e8e467998876667b779e462" member_type: VOTER last_known_addr { host: "127.10.198.1" port: 44565 } }
I20260925 22:48:40.561923 29144 catalog_manager.cc:5495] T be3c13113bcd4f36824fcc41551805e8 P cd7662f07e8e467998876667b779e462 reported cstate change: term changed from 0 to 1, leader changed from <none> to cd7662f07e8e467998876667b779e462 (127.10.198.1). New cstate: current_term: 1 leader_uuid: "cd7662f07e8e467998876667b779e462" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4733da8ac5794208a97cb3a2c063779c" member_type: VOTER last_known_addr { host: "127.10.198.3" port: 39625 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7c9c6850bc254babbd437f31fed53ff0" member_type: VOTER last_known_addr { host: "127.10.198.2" port: 46119 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cd7662f07e8e467998876667b779e462" member_type: VOTER last_known_addr { host: "127.10.198.1" port: 44565 } health_report { overall_health: HEALTHY } } }
W20260925 22:48:40.589721 29608 tablet.cc:2369] T be3c13113bcd4f36824fcc41551805e8 P 4733da8ac5794208a97cb3a2c063779c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 22:48:40.601787 29620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:48:40.597751 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41430 (local address 127.10.198.1:44565)
0925 22:48:40.597923 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0925 22:48:40.597928 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:48:40.597945 (+ 17us) server_negotiation.cc:407] Connection header received
0925 22:48:40.598143 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:48:40.598145 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:48:40.598206 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:48:40.598329 (+ 123us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 22:48:40.598586 (+ 257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:48:40.599121 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:48:40.600112 (+ 991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:48:40.600346 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:48:40.600464 (+ 118us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:48:40.600815 (+ 351us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 22:48:40.601266 (+ 451us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 22:48:40.601437 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0925 22:48:40.601565 (+ 128us) server_negotiation.cc:299] Negotiation successful
0925 22:48:40.601614 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
W20260925 22:48:40.601886 29474 tablet.cc:2369] T be3c13113bcd4f36824fcc41551805e8 P 7c9c6850bc254babbd437f31fed53ff0: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 22:48:40.608953 29427 raft_consensus.cc:1261] T be3c13113bcd4f36824fcc41551805e8 P 7c9c6850bc254babbd437f31fed53ff0 [term 1 FOLLOWER]: Refusing update from remote peer cd7662f07e8e467998876667b779e462: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 22:48:40.608952 29561 raft_consensus.cc:1261] T be3c13113bcd4f36824fcc41551805e8 P 4733da8ac5794208a97cb3a2c063779c [term 1 FOLLOWER]: Refusing update from remote peer cd7662f07e8e467998876667b779e462: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 22:48:40.609867 29635 consensus_queue.cc:1035] T be3c13113bcd4f36824fcc41551805e8 P cd7662f07e8e467998876667b779e462 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4733da8ac5794208a97cb3a2c063779c" member_type: VOTER last_known_addr { host: "127.10.198.3" port: 39625 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 22:48:40.610119 29632 consensus_queue.cc:1035] T be3c13113bcd4f36824fcc41551805e8 P cd7662f07e8e467998876667b779e462 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7c9c6850bc254babbd437f31fed53ff0" member_type: VOTER last_known_addr { host: "127.10.198.2" port: 46119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 22:48:40.620177 29640 mvcc.cc:205] Tried to move back new op lower bound from 7333382228404977664 to 7333382228209422336. Current Snapshot: MvccSnapshot[applied={T|T < 7333382228404977664}]
I20260925 22:48:40.630528 29144 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36312:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 22:48:40.630708 29144 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36312:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 22:48:40.633627 29144 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e452e50fa17e4ebb948423eb256aeeb5: Sending DeleteTablet for 3 replicas of tablet be3c13113bcd4f36824fcc41551805e8
I20260925 22:48:40.634663 29541 tablet_service.cc:1514] Processing DeleteTablet for tablet be3c13113bcd4f36824fcc41551805e8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 22:48:40 UTC) from {username='kudu', principal='kudu/127.10.198.62@KRBTEST.COM'} at 127.0.0.1:52826
I20260925 22:48:40.634671 29407 tablet_service.cc:1514] Processing DeleteTablet for tablet be3c13113bcd4f36824fcc41551805e8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 22:48:40 UTC) from {username='kudu', principal='kudu/127.10.198.62@KRBTEST.COM'} at 127.0.0.1:39668
I20260925 22:48:40.635283 29648 tablet_replica.cc:330] T be3c13113bcd4f36824fcc41551805e8 P 4733da8ac5794208a97cb3a2c063779c: stopping tablet replica
I20260925 22:48:40.635627 29273 tablet_service.cc:1514] Processing DeleteTablet for tablet be3c13113bcd4f36824fcc41551805e8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 22:48:40 UTC) from {username='kudu', principal='kudu/127.10.198.62@KRBTEST.COM'} at 127.0.0.1:41422
I20260925 22:48:40.635865 29648 raft_consensus.cc:2229] T be3c13113bcd4f36824fcc41551805e8 P 4733da8ac5794208a97cb3a2c063779c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 22:48:40.636235 29650 tablet_replica.cc:330] T be3c13113bcd4f36824fcc41551805e8 P cd7662f07e8e467998876667b779e462: stopping tablet replica
I20260925 22:48:40.636348 11032 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrWXm8X/build/debug/bin/kudu with pid 29210
I20260925 22:48:40.636443 29648 raft_consensus.cc:2258] T be3c13113bcd4f36824fcc41551805e8 P 4733da8ac5794208a97cb3a2c063779c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 22:48:40.639650 29648 ts_tablet_manager.cc:1905] T be3c13113bcd4f36824fcc41551805e8 P 4733da8ac5794208a97cb3a2c063779c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 22:48:40.640116 29649 tablet_replica.cc:330] T be3c13113bcd4f36824fcc41551805e8 P 7c9c6850bc254babbd437f31fed53ff0: stopping tablet replica
I20260925 22:48:40.640496 29649 raft_consensus.cc:2229] T be3c13113bcd4f36824fcc41551805e8 P 7c9c6850bc254babbd437f31fed53ff0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 22:48:40.640884 29649 raft_consensus.cc:2258] T be3c13113bcd4f36824fcc41551805e8 P 7c9c6850bc254babbd437f31fed53ff0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 22:48:40.642779 29649 ts_tablet_manager.cc:1905] T be3c13113bcd4f36824fcc41551805e8 P 7c9c6850bc254babbd437f31fed53ff0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 22:48:40.642936 29648 ts_tablet_manager.cc:1918] T be3c13113bcd4f36824fcc41551805e8 P 4733da8ac5794208a97cb3a2c063779c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 22:48:40.643043 29648 log.cc:1196] T be3c13113bcd4f36824fcc41551805e8 P 4733da8ac5794208a97cb3a2c063779c: Deleting WAL directory at /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790376259650642-11032-0/minicluster-data/ts-2/wal/wals/be3c13113bcd4f36824fcc41551805e8
I20260925 22:48:40.643435 29648 ts_tablet_manager.cc:1939] T be3c13113bcd4f36824fcc41551805e8 P 4733da8ac5794208a97cb3a2c063779c: Deleting consensus metadata
W20260925 22:48:40.644071 29132 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.10.198.1:44565 (error 108)
I20260925 22:48:40.644358 11032 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrWXm8X/build/debug/bin/kudu with pid 29344
I20260925 22:48:40.645679 29135 catalog_manager.cc:4841] TS 4733da8ac5794208a97cb3a2c063779c (127.10.198.3:39625): tablet be3c13113bcd4f36824fcc41551805e8 (table test-table [id=df8d84e430654bda90b062e817e7e53e]) successfully deleted
I20260925 22:48:40.646011 29623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:48:40.645620 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.198.1:44565 (local address 127.0.0.1:41442)
0925 22:48:40.645795 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0925 22:48:40.645893 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.198.1:44565: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
W20260925 22:48:40.646231 29132 catalog_manager.cc:4584] TS cd7662f07e8e467998876667b779e462 (127.10.198.1:44565): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet be3c13113bcd4f36824fcc41551805e8: Network error: Client connection negotiation failed: client connection to 127.10.198.1:44565: connect: Connection refused (error 111)
I20260925 22:48:40.651283 11032 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrWXm8X/build/debug/bin/kudu with pid 29478
I20260925 22:48:40.653307 29623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:48:40.651529 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.198.2:46119 (local address 127.0.0.1:39678)
0925 22:48:40.653072 (+ 1543us) negotiation.cc:107] Waiting for socket to connect
0925 22:48:40.653169 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.198.2:46119: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1404}
I20260925 22:48:40.657693 11032 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrWXm8X/build/debug/bin/kudu with pid 29116
2026-09-25T22:48:40Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (2511 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-09-25T22:48:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T22:48:40Z Disabled control of system clock
I20260925 22:48:40.689891 11032 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrWXm8X/build/debug/bin/kudu
/tmp/dist-test-taskrWXm8X/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790376259650642-11032-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790376259650642-11032-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790376259650642-11032-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790376259650642-11032-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.198.62:33247
--webserver_interface=127.10.198.62
--webserver_port=0
--builtin_ntp_servers=127.10.198.20:34965
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.198.62:33247
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260925 22:48:40.811062 29655 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:48:40.811401 29655 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:48:40.811441 29655 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:48:40.816228 29655 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 22:48:40.816308 29655 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:48:40.816324 29655 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:48:40.816344 29655 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 22:48:40.816360 29655 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 22:48:40.821681 29655 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.198.20:34965
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790376259650642-11032-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790376259650642-11032-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.198.62:33247
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.198.62:33247
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790376259650642-11032-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.198.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29655
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790376259650642-11032-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:09:16 UTC on ccycloud.quasar-afmhyw.root.comops.site
build id 4673379
I20260925 22:48:40.822890 29655 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:48:40.824497 29655 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:48:40.833845 29660 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:48:40.833845 29663 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:48:40.833845 29661 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:48:40.834367 29655 server_base.cc:1034] running on GCE node
I20260925 22:48:40.834985 29655 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:48:40.836331 29655 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:48:40.837527 29655 hybrid_clock.cc:648] HybridClock initialized: now 1790376520837510 us; error 46 us; skew 500 ppm
I20260925 22:48:40.839913 29655 webserver.cc:460] Webserver started at http://127.10.198.62:44331/ using document root <none> and password file <none>
I20260925 22:48:40.840845 29655 fs_manager.cc:359] Metadata directory not provided
I20260925 22:48:40.840946 29655 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:48:40.841189 29655 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:48:40.843605 29655 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790376259650642-11032-0/minicluster-data/master-0/data/instance:
uuid: "ed0963ae2e61483bbe3719bab08b719f"
format_stamp: "Formatted at 2026-09-25 22:48:40 on dist-test-slave-5vdr"
server_key: "5625f4bec719de49b32f8eee6c42aabf"
server_key_iv: "d70912346f106dfbce5089e47af4011b"
server_key_version: "encryptionkey@0"
I20260925 22:48:40.844414 29655 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790376259650642-11032-0/minicluster-data/master-0/wal/instance:
uuid: "ed0963ae2e61483bbe3719bab08b719f"
format_stamp: "Formatted at 2026-09-25 22:48:40 on dist-test-slave-5vdr"
server_key: "5625f4bec719de49b32f8eee6c42aabf"
server_key_iv: "d70912346f106dfbce5089e47af4011b"
server_key_version: "encryptionkey@0"
I20260925 22:48:40.848773 29655 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260925 22:48:40.851792 29669 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:48:40.853327 29655 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 22:48:40.853549 29655 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790376259650642-11032-0/minicluster-data/master-0/data,/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790376259650642-11032-0/minicluster-data/master-0/wal
uuid: "ed0963ae2e61483bbe3719bab08b719f"
format_stamp: "Formatted at 2026-09-25 22:48:40 on dist-test-slave-5vdr"
server_key: "5625f4bec719de49b32f8eee6c42aabf"
server_key_iv: "d70912346f106dfbce5089e47af4011b"
server_key_version: "encryptionkey@0"
I20260925 22:48:40.853713 29655 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790376259650642-11032-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790376259650642-11032-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790376259650642-11032-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:48:40.870457 29655 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:48:40.871299 29655 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:48:40.871552 29655 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:48:40.879374 29655 rpc_server.cc:307] RPC server started. Bound to: 127.10.198.62:33247
I20260925 22:48:40.879393 29721 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.198.62:33247 every 8 connection(s)
I20260925 22:48:40.880533 29655 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790376259650642-11032-0/minicluster-data/master-0/data/info.pb
I20260925 22:48:40.883977 29722 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:48:40.887115 11032 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrWXm8X/build/debug/bin/kudu as pid 29655
I20260925 22:48:40.887290 11032 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790376259650642-11032-0/minicluster-data/master-0/wal/instance
I20260925 22:48:40.887589 11032 external_mini_cluster.cc:1428] Setting key 7c0fde94ed33f4639905a4c446688095
I20260925 22:48:40.890650 29722 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ed0963ae2e61483bbe3719bab08b719f: Bootstrap starting.
I20260925 22:48:40.893169 29722 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ed0963ae2e61483bbe3719bab08b719f: Neither blocks nor log segments found. Creating new log.
I20260925 22:48:40.894177 29722 log.cc:824] T 00000000000000000000000000000000 P ed0963ae2e61483bbe3719bab08b719f: Log is configured to *not* fsync() on all Append() calls
I20260925 22:48:40.896375 29722 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ed0963ae2e61483bbe3719bab08b719f: No bootstrap required, opened a new log
I20260925 22:48:40.897841 29725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:48:40.889116 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38030 (local address 127.10.198.62:33247)
0925 22:48:40.889851 (+ 735us) server_negotiation.cc:206] Beginning negotiation
0925 22:48:40.889868 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0925 22:48:40.889907 (+ 39us) server_negotiation.cc:407] Connection header received
0925 22:48:40.890560 (+ 653us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:48:40.890578 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:48:40.891087 (+ 509us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:48:40.891462 (+ 375us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:48:40.892005 (+ 543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:48:40.893031 (+ 1026us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:48:40.894186 (+ 1155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:48:40.894779 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:48:40.894969 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:48:40.895940 (+ 971us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:48:40.895946 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:48:40.895961 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 22:48:40.896373 (+ 412us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:48:40.896431 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 22:48:40.896452 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:48:40.896575 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0925 22:48:40.896765 (+ 190us) server_negotiation.cc:299] Negotiation successful
0925 22:48:40.897075 (+ 310us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":352,"thread_start_us":162,"threads_started":1}
I20260925 22:48:40.900283 29722 raft_consensus.cc:348] T 00000000000000000000000000000000 P ed0963ae2e61483bbe3719bab08b719f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed0963ae2e61483bbe3719bab08b719f" member_type: VOTER last_known_addr { host: "127.10.198.62" port: 33247 } }
I20260925 22:48:40.900564 29722 raft_consensus.cc:374] T 00000000000000000000000000000000 P ed0963ae2e61483bbe3719bab08b719f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:48:40.900776 29722 raft_consensus.cc:729] T 00000000000000000000000000000000 P ed0963ae2e61483bbe3719bab08b719f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ed0963ae2e61483bbe3719bab08b719f, State: Initialized, Role: FOLLOWER
I20260925 22:48:40.901346 29722 consensus_queue.cc:260] T 00000000000000000000000000000000 P ed0963ae2e61483bbe3719bab08b719f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed0963ae2e61483bbe3719bab08b719f" member_type: VOTER last_known_addr { host: "127.10.198.62" port: 33247 } }
I20260925 22:48:40.901536 29722 raft_consensus.cc:388] T 00000000000000000000000000000000 P ed0963ae2e61483bbe3719bab08b719f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 22:48:40.901631 29722 raft_consensus.cc:482] T 00000000000000000000000000000000 P ed0963ae2e61483bbe3719bab08b719f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 22:48:40.901741 29722 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ed0963ae2e61483bbe3719bab08b719f [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:48:40.902865 29722 raft_consensus.cc:504] T 00000000000000000000000000000000 P ed0963ae2e61483bbe3719bab08b719f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed0963ae2e61483bbe3719bab08b719f" member_type: VOTER last_known_addr { host: "127.10.198.62" port: 33247 } }
I20260925 22:48:40.903318 29722 leader_election.cc:302] T 00000000000000000000000000000000 P ed0963ae2e61483bbe3719bab08b719f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ed0963ae2e61483bbe3719bab08b719f; no voters:
I20260925 22:48:40.903717 29722 leader_election.cc:288] T 00000000000000000000000000000000 P ed0963ae2e61483bbe3719bab08b719f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 22:48:40.903889 29727 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ed0963ae2e61483bbe3719bab08b719f [term 1 FOLLOWER]: Leader election won for term 1
I20260925 22:48:40.904155 29727 raft_consensus.cc:686] T 00000000000000000000000000000000 P ed0963ae2e61483bbe3719bab08b719f [term 1 LEADER]: Becoming Leader. State: Replica: ed0963ae2e61483bbe3719bab08b719f, State: Running, Role: LEADER
I20260925 22:48:40.904495 29727 consensus_queue.cc:237] T 00000000000000000000000000000000 P ed0963ae2e61483bbe3719bab08b719f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed0963ae2e61483bbe3719bab08b719f" member_type: VOTER last_known_addr { host: "127.10.198.62" port: 33247 } }
I20260925 22:48:40.905165 29722 sys_catalog.cc:564] T 00000000000000000000000000000000 P ed0963ae2e61483bbe3719bab08b719f [sys.catalog]: configured and running, proceeding with master startup.
I20260925 22:48:40.906270 29728 sys_catalog.cc:455] T 00000000000000000000000000000000 P ed0963ae2e61483bbe3719bab08b719f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ed0963ae2e61483bbe3719bab08b719f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed0963ae2e61483bbe3719bab08b719f" member_type: VOTER last_known_addr { host: "127.10.198.62" port: 33247 } } }
I20260925 22:48:40.906419 29728 sys_catalog.cc:458] T 00000000000000000000000000000000 P ed0963ae2e61483bbe3719bab08b719f [sys.catalog]: This master's current role is: LEADER
I20260925 22:48:40.906893 29736 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 22:48:40.907476 29729 sys_catalog.cc:455] T 00000000000000000000000000000000 P ed0963ae2e61483bbe3719bab08b719f [sys.catalog]: SysCatalogTable state changed. Reason: New leader ed0963ae2e61483bbe3719bab08b719f. Latest consensus state: current_term: 1 leader_uuid: "ed0963ae2e61483bbe3719bab08b719f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed0963ae2e61483bbe3719bab08b719f" member_type: VOTER last_known_addr { host: "127.10.198.62" port: 33247 } } }
I20260925 22:48:40.907605 29729 sys_catalog.cc:458] T 00000000000000000000000000000000 P ed0963ae2e61483bbe3719bab08b719f [sys.catalog]: This master's current role is: LEADER
I20260925 22:48:40.910166 29736 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 22:48:40.917101 29736 catalog_manager.cc:1347] Generated new cluster ID: f9cd4aa4e6a04cd785e400f1ad0523ec
I20260925 22:48:40.917209 29736 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 22:48:40.925505 29736 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 22:48:40.926869 29736 catalog_manager.cc:1504] Loading token signing keys...
I20260925 22:48:40.939253 29736 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ed0963ae2e61483bbe3719bab08b719f: Generated new TSK 0
I20260925 22:48:40.940317 29736 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 22:48:40.950148 11032 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrWXm8X/build/debug/bin/kudu
/tmp/dist-test-taskrWXm8X/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790376259650642-11032-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790376259650642-11032-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790376259650642-11032-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790376259650642-11032-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.198.1:0
--local_ip_for_outbound_sockets=127.10.198.1
--webserver_interface=127.10.198.1
--webserver_port=0
--tserver_master_addrs=127.10.198.62:33247
--builtin_ntp_servers=127.10.198.20:34965
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260925 22:48:41.074158 29746 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:48:41.074544 29746 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:48:41.074626 29746 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:48:41.079155 29746 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:48:41.079295 29746 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:48:41.079404 29746 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.198.1
I20260925 22:48:41.085628 29746 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.198.20:34965
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790376259650642-11032-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790376259650642-11032-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.198.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790376259650642-11032-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.198.1
--webserver_port=0
--tserver_master_addrs=127.10.198.62:33247
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29746
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.198.1
--log_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790376259650642-11032-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:09:16 UTC on ccycloud.quasar-afmhyw.root.comops.site
build id 4673379
I20260925 22:48:41.086947 29746 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:48:41.088519 29746 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:48:41.100158 29754 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:48:41.100189 29752 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:48:41.100158 29751 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:48:41.100580 29746 server_base.cc:1034] running on GCE node
I20260925 22:48:41.101117 29746 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:48:41.101908 29746 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:48:41.103137 29746 hybrid_clock.cc:648] HybridClock initialized: now 1790376521103123 us; error 38 us; skew 500 ppm
I20260925 22:48:41.105710 29746 webserver.cc:460] Webserver started at http://127.10.198.1:34927/ using document root <none> and password file <none>
I20260925 22:48:41.106451 29746 fs_manager.cc:359] Metadata directory not provided
I20260925 22:48:41.106560 29746 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:48:41.106848 29746 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:48:41.109319 29746 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790376259650642-11032-0/minicluster-data/ts-0/data/instance:
uuid: "14fc637d4eca40aabc7cb4d8ff2e1d59"
format_stamp: "Formatted at 2026-09-25 22:48:41 on dist-test-slave-5vdr"
server_key: "639922325c21a8e0e7fb9411f1d6efef"
server_key_iv: "a9abe91e55f5b353441d0624ead8bd2d"
server_key_version: "encryptionkey@0"
I20260925 22:48:41.110025 29746 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790376259650642-11032-0/minicluster-data/ts-0/wal/instance:
uuid: "14fc637d4eca40aabc7cb4d8ff2e1d59"
format_stamp: "Formatted at 2026-09-25 22:48:41 on dist-test-slave-5vdr"
server_key: "639922325c21a8e0e7fb9411f1d6efef"
server_key_iv: "a9abe91e55f5b353441d0624ead8bd2d"
server_key_version: "encryptionkey@0"
I20260925 22:48:41.114894 29746 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260925 22:48:41.118197 29760 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:48:41.119769 29746 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260925 22:48:41.120051 29746 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790376259650642-11032-0/minicluster-data/ts-0/data,/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790376259650642-11032-0/minicluster-data/ts-0/wal
uuid: "14fc637d4eca40aabc7cb4d8ff2e1d59"
format_stamp: "Formatted at 2026-09-25 22:48:41 on dist-test-slave-5vdr"
server_key: "639922325c21a8e0e7fb9411f1d6efef"
server_key_iv: "a9abe91e55f5b353441d0624ead8bd2d"
server_key_version: "encryptionkey@0"
I20260925 22:48:41.120219 29746 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790376259650642-11032-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790376259650642-11032-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790376259650642-11032-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:48:41.145607 29746 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:48:41.146479 29746 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:48:41.146723 29746 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:48:41.147413 29746 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 22:48:41.148507 29746 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:48:41.148562 29746 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:48:41.148607 29746 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:48:41.148623 29746 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 22:48:41.158622 29746 rpc_server.cc:307] RPC server started. Bound to: 127.10.198.1:34281
I20260925 22:48:41.158666 29873 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.198.1:34281 every 8 connection(s)
I20260925 22:48:41.159650 29746 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790376259650642-11032-0/minicluster-data/ts-0/data/info.pb
I20260925 22:48:41.168113 11032 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrWXm8X/build/debug/bin/kudu as pid 29746
I20260925 22:48:41.168277 11032 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790376259650642-11032-0/minicluster-data/ts-0/wal/instance
I20260925 22:48:41.168635 11032 external_mini_cluster.cc:1428] Setting key 49b30818760b82cacdd1be3bdbfcc5c5
I20260925 22:48:41.170984 11032 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrWXm8X/build/debug/bin/kudu
/tmp/dist-test-taskrWXm8X/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790376259650642-11032-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790376259650642-11032-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790376259650642-11032-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790376259650642-11032-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.198.2:0
--local_ip_for_outbound_sockets=127.10.198.2
--webserver_interface=127.10.198.2
--webserver_port=0
--tserver_master_addrs=127.10.198.62:33247
--builtin_ntp_servers=127.10.198.20:34965
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260925 22:48:41.170985 29725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:48:41.162046 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.198.1:45791 (local address 127.10.198.62:33247)
0925 22:48:41.162194 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0925 22:48:41.162198 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:48:41.163454 (+ 1256us) server_negotiation.cc:407] Connection header received
0925 22:48:41.165242 (+ 1788us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:48:41.165245 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:48:41.165305 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:48:41.165389 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:48:41.166506 (+ 1117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:48:41.167077 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:48:41.168475 (+ 1398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:48:41.168824 (+ 349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:48:41.168960 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:48:41.170431 (+ 1471us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:48:41.170435 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:48:41.170437 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 22:48:41.170451 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:48:41.170473 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 22:48:41.170482 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:48:41.170565 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0925 22:48:41.170757 (+ 192us) server_negotiation.cc:299] Negotiation successful
0925 22:48:41.170834 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260925 22:48:41.172186 29876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:48:41.162420 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.198.62:33247 (local address 127.10.198.1:45791)
0925 22:48:41.163230 (+ 810us) negotiation.cc:107] Waiting for socket to connect
0925 22:48:41.163284 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0925 22:48:41.164938 (+ 1654us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:48:41.165607 (+ 669us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:48:41.165616 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:48:41.165722 (+ 106us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:48:41.166323 (+ 601us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:48:41.166338 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:48:41.167236 (+ 898us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:48:41.167239 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:48:41.168290 (+ 1051us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:48:41.168311 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:48:41.169011 (+ 700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:48:41.169015 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:48:41.169193 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:48:41.170024 (+ 831us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 22:48:41.170025 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:48:41.170117 (+ 92us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 22:48:41.170244 (+ 127us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:48:41.170585 (+ 341us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:48:41.170596 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:48:41.170603 (+ 7us) client_negotiation.cc:769] Sending connection context
0925 22:48:41.170757 (+ 154us) client_negotiation.cc:240] Negotiation successful
0925 22:48:41.171129 (+ 372us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":349,"thread_start_us":150,"threads_started":1}
I20260925 22:48:41.173743 29874 heartbeater.cc:347] Connected to a master server at 127.10.198.62:33247
I20260925 22:48:41.174044 29874 heartbeater.cc:464] Registering TS with master...
I20260925 22:48:41.174660 29874 heartbeater.cc:511] Master 127.10.198.62:33247 requested a full tablet report, sending...
I20260925 22:48:41.176743 29684 ts_manager.cc:194] Registered new tserver with Master: 14fc637d4eca40aabc7cb4d8ff2e1d59 (127.10.198.1:34281)
I20260925 22:48:41.178746 29684 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.10.198.1:45791
W20260925 22:48:41.290637 29877 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:48:41.290954 29877 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:48:41.290997 29877 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:48:41.295275 29877 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:48:41.295360 29877 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:48:41.295419 29877 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.198.2
I20260925 22:48:41.300856 29877 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.198.20:34965
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790376259650642-11032-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790376259650642-11032-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.198.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790376259650642-11032-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.198.2
--webserver_port=0
--tserver_master_addrs=127.10.198.62:33247
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29877
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.198.2
--log_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790376259650642-11032-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:09:16 UTC on ccycloud.quasar-afmhyw.root.comops.site
build id 4673379
I20260925 22:48:41.302110 29877 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:48:41.303593 29877 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:48:41.314416 29885 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:48:41.314420 29882 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:48:41.314445 29883 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:48:41.314563 29877 server_base.cc:1034] running on GCE node
I20260925 22:48:41.315236 29877 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:48:41.316028 29877 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:48:41.317232 29877 hybrid_clock.cc:648] HybridClock initialized: now 1790376521317213 us; error 34 us; skew 500 ppm
I20260925 22:48:41.319371 29877 webserver.cc:460] Webserver started at http://127.10.198.2:43403/ using document root <none> and password file <none>
I20260925 22:48:41.320120 29877 fs_manager.cc:359] Metadata directory not provided
I20260925 22:48:41.320228 29877 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:48:41.320480 29877 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:48:41.322779 29877 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790376259650642-11032-0/minicluster-data/ts-1/data/instance:
uuid: "b22bee1e2c3a425985653421554cadfa"
format_stamp: "Formatted at 2026-09-25 22:48:41 on dist-test-slave-5vdr"
server_key: "6b31b498972df672194afc50d7a0a60f"
server_key_iv: "cb5d75b91c9f0f223014c924de22cddf"
server_key_version: "encryptionkey@0"
I20260925 22:48:41.323426 29877 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790376259650642-11032-0/minicluster-data/ts-1/wal/instance:
uuid: "b22bee1e2c3a425985653421554cadfa"
format_stamp: "Formatted at 2026-09-25 22:48:41 on dist-test-slave-5vdr"
server_key: "6b31b498972df672194afc50d7a0a60f"
server_key_iv: "cb5d75b91c9f0f223014c924de22cddf"
server_key_version: "encryptionkey@0"
I20260925 22:48:41.327960 29877 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.001s
I20260925 22:48:41.330742 29891 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:48:41.332243 29877 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 22:48:41.332418 29877 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790376259650642-11032-0/minicluster-data/ts-1/data,/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790376259650642-11032-0/minicluster-data/ts-1/wal
uuid: "b22bee1e2c3a425985653421554cadfa"
format_stamp: "Formatted at 2026-09-25 22:48:41 on dist-test-slave-5vdr"
server_key: "6b31b498972df672194afc50d7a0a60f"
server_key_iv: "cb5d75b91c9f0f223014c924de22cddf"
server_key_version: "encryptionkey@0"
I20260925 22:48:41.332563 29877 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790376259650642-11032-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790376259650642-11032-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790376259650642-11032-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:48:41.367687 29877 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:48:41.368592 29877 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:48:41.368819 29877 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:48:41.369453 29877 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 22:48:41.370850 29877 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:48:41.370950 29877 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:48:41.371038 29877 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:48:41.371093 29877 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 22:48:41.380903 29877 rpc_server.cc:307] RPC server started. Bound to: 127.10.198.2:37035
I20260925 22:48:41.380923 30004 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.198.2:37035 every 8 connection(s)
I20260925 22:48:41.381925 29877 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790376259650642-11032-0/minicluster-data/ts-1/data/info.pb
I20260925 22:48:41.388857 11032 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrWXm8X/build/debug/bin/kudu as pid 29877
I20260925 22:48:41.389036 11032 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790376259650642-11032-0/minicluster-data/ts-1/wal/instance
I20260925 22:48:41.389362 11032 external_mini_cluster.cc:1428] Setting key 411b9eb2bd07dc583360d67afd8a8c25
I20260925 22:48:41.391808 11032 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrWXm8X/build/debug/bin/kudu
/tmp/dist-test-taskrWXm8X/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790376259650642-11032-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790376259650642-11032-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790376259650642-11032-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790376259650642-11032-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.198.3:0
--local_ip_for_outbound_sockets=127.10.198.3
--webserver_interface=127.10.198.3
--webserver_port=0
--tserver_master_addrs=127.10.198.62:33247
--builtin_ntp_servers=127.10.198.20:34965
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260925 22:48:41.393004 29725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:48:41.384012 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.198.2:38715 (local address 127.10.198.62:33247)
0925 22:48:41.384185 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0925 22:48:41.384189 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:48:41.385460 (+ 1271us) server_negotiation.cc:407] Connection header received
0925 22:48:41.387105 (+ 1645us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:48:41.387109 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:48:41.387175 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:48:41.387289 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:48:41.388340 (+ 1051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:48:41.388884 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:48:41.390506 (+ 1622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:48:41.390842 (+ 336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:48:41.390956 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:48:41.392446 (+ 1490us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:48:41.392449 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:48:41.392451 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 22:48:41.392465 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:48:41.392486 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 22:48:41.392495 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:48:41.392585 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0925 22:48:41.392778 (+ 193us) server_negotiation.cc:299] Negotiation successful
0925 22:48:41.392852 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260925 22:48:41.394057 30007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:48:41.384450 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.198.62:33247 (local address 127.10.198.2:38715)
0925 22:48:41.385289 (+ 839us) negotiation.cc:107] Waiting for socket to connect
0925 22:48:41.385338 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0925 22:48:41.386838 (+ 1500us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:48:41.387490 (+ 652us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:48:41.387497 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:48:41.387601 (+ 104us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:48:41.388169 (+ 568us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:48:41.388183 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:48:41.389046 (+ 863us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:48:41.389050 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:48:41.390342 (+ 1292us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:48:41.390356 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:48:41.391063 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:48:41.391068 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:48:41.391283 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:48:41.392083 (+ 800us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 22:48:41.392087 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:48:41.392156 (+ 69us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 22:48:41.392248 (+ 92us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:48:41.392589 (+ 341us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:48:41.392602 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:48:41.392611 (+ 9us) client_negotiation.cc:769] Sending connection context
0925 22:48:41.392767 (+ 156us) client_negotiation.cc:240] Negotiation successful
0925 22:48:41.393122 (+ 355us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":321,"thread_start_us":132,"threads_started":1}
I20260925 22:48:41.395460 30005 heartbeater.cc:347] Connected to a master server at 127.10.198.62:33247
I20260925 22:48:41.395744 30005 heartbeater.cc:464] Registering TS with master...
I20260925 22:48:41.396420 30005 heartbeater.cc:511] Master 127.10.198.62:33247 requested a full tablet report, sending...
I20260925 22:48:41.397596 29684 ts_manager.cc:194] Registered new tserver with Master: b22bee1e2c3a425985653421554cadfa (127.10.198.2:37035)
I20260925 22:48:41.398217 29684 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.10.198.2:38715
W20260925 22:48:41.511035 30008 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:48:41.511344 30008 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:48:41.511421 30008 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:48:41.518599 30008 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:48:41.518714 30008 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:48:41.518821 30008 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.198.3
I20260925 22:48:41.524910 30008 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.198.20:34965
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790376259650642-11032-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790376259650642-11032-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.198.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790376259650642-11032-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.198.3
--webserver_port=0
--tserver_master_addrs=127.10.198.62:33247
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30008
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.198.3
--log_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790376259650642-11032-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:09:16 UTC on ccycloud.quasar-afmhyw.root.comops.site
build id 4673379
I20260925 22:48:41.526211 30008 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:48:41.527702 30008 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:48:41.538195 30014 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:48:41.538173 30013 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:48:41.538195 30016 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:48:41.538669 30008 server_base.cc:1034] running on GCE node
I20260925 22:48:41.539140 30008 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:48:41.539933 30008 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:48:41.541136 30008 hybrid_clock.cc:648] HybridClock initialized: now 1790376521541126 us; error 39 us; skew 500 ppm
I20260925 22:48:41.543356 30008 webserver.cc:460] Webserver started at http://127.10.198.3:44585/ using document root <none> and password file <none>
I20260925 22:48:41.544134 30008 fs_manager.cc:359] Metadata directory not provided
I20260925 22:48:41.544253 30008 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:48:41.544503 30008 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:48:41.546739 30008 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790376259650642-11032-0/minicluster-data/ts-2/data/instance:
uuid: "98c4f94b242d4f3798a030f1d3b6420b"
format_stamp: "Formatted at 2026-09-25 22:48:41 on dist-test-slave-5vdr"
server_key: "f89220f4dc582b276be88ab899adf014"
server_key_iv: "050bc5df2e65fc3ea210a27088f678c9"
server_key_version: "encryptionkey@0"
I20260925 22:48:41.547369 30008 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790376259650642-11032-0/minicluster-data/ts-2/wal/instance:
uuid: "98c4f94b242d4f3798a030f1d3b6420b"
format_stamp: "Formatted at 2026-09-25 22:48:41 on dist-test-slave-5vdr"
server_key: "f89220f4dc582b276be88ab899adf014"
server_key_iv: "050bc5df2e65fc3ea210a27088f678c9"
server_key_version: "encryptionkey@0"
I20260925 22:48:41.551455 30008 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.001s
I20260925 22:48:41.554396 30022 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:48:41.555828 30008 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260925 22:48:41.556087 30008 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790376259650642-11032-0/minicluster-data/ts-2/data,/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790376259650642-11032-0/minicluster-data/ts-2/wal
uuid: "98c4f94b242d4f3798a030f1d3b6420b"
format_stamp: "Formatted at 2026-09-25 22:48:41 on dist-test-slave-5vdr"
server_key: "f89220f4dc582b276be88ab899adf014"
server_key_iv: "050bc5df2e65fc3ea210a27088f678c9"
server_key_version: "encryptionkey@0"
I20260925 22:48:41.556236 30008 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790376259650642-11032-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790376259650642-11032-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790376259650642-11032-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:48:41.573719 30008 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:48:41.574496 30008 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:48:41.574733 30008 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:48:41.575383 30008 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 22:48:41.576476 30008 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:48:41.576553 30008 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:48:41.576629 30008 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:48:41.576671 30008 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 22:48:41.585793 30008 rpc_server.cc:307] RPC server started. Bound to: 127.10.198.3:43691
I20260925 22:48:41.585817 30135 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.198.3:43691 every 8 connection(s)
I20260925 22:48:41.586856 30008 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790376259650642-11032-0/minicluster-data/ts-2/data/info.pb
I20260925 22:48:41.589314 11032 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrWXm8X/build/debug/bin/kudu as pid 30008
I20260925 22:48:41.589438 11032 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790376259650642-11032-0/minicluster-data/ts-2/wal/instance
I20260925 22:48:41.589741 11032 external_mini_cluster.cc:1428] Setting key d2b80adef672010d41c2a092b387da3e
I20260925 22:48:41.597574 29725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:48:41.589225 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.198.3:39477 (local address 127.10.198.62:33247)
0925 22:48:41.589421 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0925 22:48:41.589428 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 22:48:41.590737 (+ 1309us) server_negotiation.cc:407] Connection header received
0925 22:48:41.592358 (+ 1621us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:48:41.592361 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:48:41.592411 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:48:41.592482 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:48:41.593438 (+ 956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:48:41.593988 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:48:41.595263 (+ 1275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:48:41.595495 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:48:41.595612 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:48:41.596986 (+ 1374us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:48:41.596990 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:48:41.596992 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 22:48:41.597006 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:48:41.597027 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 22:48:41.597036 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:48:41.597104 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0925 22:48:41.597343 (+ 239us) server_negotiation.cc:299] Negotiation successful
0925 22:48:41.597413 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260925 22:48:41.598558 30138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:48:41.589649 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.198.62:33247 (local address 127.10.198.3:39477)
0925 22:48:41.590550 (+ 901us) negotiation.cc:107] Waiting for socket to connect
0925 22:48:41.590614 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0925 22:48:41.592125 (+ 1511us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:48:41.592636 (+ 511us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:48:41.592643 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:48:41.592742 (+ 99us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:48:41.593288 (+ 546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:48:41.593302 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:48:41.594122 (+ 820us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:48:41.594125 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:48:41.595099 (+ 974us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:48:41.595110 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:48:41.595712 (+ 602us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:48:41.595715 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:48:41.595977 (+ 262us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:48:41.596655 (+ 678us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 22:48:41.596656 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:48:41.596707 (+ 51us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 22:48:41.596810 (+ 103us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:48:41.597150 (+ 340us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:48:41.597160 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:48:41.597166 (+ 6us) client_negotiation.cc:769] Sending connection context
0925 22:48:41.597295 (+ 129us) client_negotiation.cc:240] Negotiation successful
0925 22:48:41.597574 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":387,"thread_start_us":213,"threads_started":1}
I20260925 22:48:41.600000 30136 heartbeater.cc:347] Connected to a master server at 127.10.198.62:33247
I20260925 22:48:41.600301 30136 heartbeater.cc:464] Registering TS with master...
I20260925 22:48:41.600936 30136 heartbeater.cc:511] Master 127.10.198.62:33247 requested a full tablet report, sending...
I20260925 22:48:41.602207 29684 ts_manager.cc:194] Registered new tserver with Master: 98c4f94b242d4f3798a030f1d3b6420b (127.10.198.3:43691)
I20260925 22:48:41.602871 29684 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.10.198.3:39477
I20260925 22:48:41.604094 11032 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 22:48:41.610877 29725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:48:41.606542 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38196 (local address 127.10.198.62:33247)
0925 22:48:41.606737 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0925 22:48:41.606742 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:48:41.606890 (+ 148us) server_negotiation.cc:407] Connection header received
0925 22:48:41.607086 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:48:41.607089 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:48:41.607143 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:48:41.607230 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:48:41.607554 (+ 324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:48:41.608174 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:48:41.609055 (+ 881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:48:41.609293 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:48:41.609408 (+ 115us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:48:41.610344 (+ 936us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:48:41.610347 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:48:41.610350 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 22:48:41.610366 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:48:41.610391 (+ 25us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 22:48:41.610399 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:48:41.610531 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0925 22:48:41.610659 (+ 128us) server_negotiation.cc:299] Negotiation successful
0925 22:48:41.610712 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260925 22:48:41.614698 29684 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:38196:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 22:48:41.618432 29684 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 22:48:41.637118 30151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:48:41.630812 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.198.2:37035 (local address 127.0.0.1:43606)
0925 22:48:41.631135 (+ 323us) negotiation.cc:107] Waiting for socket to connect
0925 22:48:41.631149 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 22:48:41.631245 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:48:41.632949 (+ 1704us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:48:41.632953 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:48:41.632974 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:48:41.633083 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:48:41.633090 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:48:41.634286 (+ 1196us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:48:41.634292 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:48:41.635399 (+ 1107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:48:41.635410 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:48:41.636425 (+ 1015us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:48:41.636429 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:48:41.636570 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:48:41.636612 (+ 42us) client_negotiation.cc:769] Sending connection context
0925 22:48:41.636789 (+ 177us) client_negotiation.cc:240] Negotiation successful
0925 22:48:41.636941 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":258,"mutex_wait_us":5,"thread_start_us":137,"threads_started":1}
I20260925 22:48:41.637121 30152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:48:41.631517 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43606 (local address 127.10.198.2:37035)
0925 22:48:41.632339 (+ 822us) server_negotiation.cc:206] Beginning negotiation
0925 22:48:41.632356 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0925 22:48:41.632383 (+ 27us) server_negotiation.cc:407] Connection header received
0925 22:48:41.632462 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:48:41.632467 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:48:41.632677 (+ 210us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:48:41.632843 (+ 166us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:48:41.633221 (+ 378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:48:41.634151 (+ 930us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:48:41.635514 (+ 1363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:48:41.636272 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:48:41.636403 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:48:41.636634 (+ 231us) server_negotiation.cc:1035] Waiting for connection context
0925 22:48:41.636828 (+ 194us) server_negotiation.cc:299] Negotiation successful
0925 22:48:41.636941 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":699,"thread_start_us":111,"threads_started":1}
I20260925 22:48:41.637545 30149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:48:41.630405 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35706 (local address 127.10.198.3:43691)
0925 22:48:41.631126 (+ 721us) server_negotiation.cc:206] Beginning negotiation
0925 22:48:41.631136 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 22:48:41.631161 (+ 25us) server_negotiation.cc:407] Connection header received
0925 22:48:41.631247 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:48:41.631254 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:48:41.631541 (+ 287us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:48:41.631713 (+ 172us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:48:41.632453 (+ 740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:48:41.633611 (+ 1158us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:48:41.635448 (+ 1837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:48:41.636420 (+ 972us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:48:41.636548 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:48:41.636750 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0925 22:48:41.637218 (+ 468us) server_negotiation.cc:299] Negotiation successful
0925 22:48:41.637340 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":617,"thread_start_us":164,"threads_started":1}
I20260925 22:48:41.637603 30150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:48:41.630539 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58564 (local address 127.10.198.1:34281)
0925 22:48:41.630928 (+ 389us) server_negotiation.cc:206] Beginning negotiation
0925 22:48:41.630936 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 22:48:41.630966 (+ 30us) server_negotiation.cc:407] Connection header received
0925 22:48:41.631364 (+ 398us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:48:41.631373 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:48:41.631604 (+ 231us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:48:41.631751 (+ 147us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:48:41.632447 (+ 696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:48:41.633477 (+ 1030us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:48:41.635582 (+ 2105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:48:41.636480 (+ 898us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:48:41.636564 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:48:41.636750 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0925 22:48:41.637313 (+ 563us) server_negotiation.cc:299] Negotiation successful
0925 22:48:41.637424 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":283,"thread_start_us":61,"threads_started":1}
I20260925 22:48:41.638104 30148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:48:41.629544 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.198.1:34281 (local address 127.0.0.1:58564)
0925 22:48:41.630811 (+ 1267us) negotiation.cc:107] Waiting for socket to connect
0925 22:48:41.630827 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 22:48:41.631163 (+ 336us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:48:41.631996 (+ 833us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:48:41.631999 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:48:41.632052 (+ 53us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:48:41.632287 (+ 235us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:48:41.632299 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:48:41.633607 (+ 1308us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:48:41.633613 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:48:41.635424 (+ 1811us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:48:41.635434 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:48:41.636873 (+ 1439us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:48:41.636877 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:48:41.637129 (+ 252us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:48:41.637168 (+ 39us) client_negotiation.cc:769] Sending connection context
0925 22:48:41.637747 (+ 579us) client_negotiation.cc:240] Negotiation successful
0925 22:48:41.637871 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1175,"mutex_wait_us":113,"spinlock_wait_cycles":49408,"thread_start_us":131,"threads_started":1}
I20260925 22:48:41.638221 30147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:48:41.629417 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.198.3:43691 (local address 127.0.0.1:35706)
0925 22:48:41.629972 (+ 555us) negotiation.cc:107] Waiting for socket to connect
0925 22:48:41.630070 (+ 98us) client_negotiation.cc:174] Beginning negotiation
0925 22:48:41.630316 (+ 246us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:48:41.631944 (+ 1628us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:48:41.631949 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:48:41.632045 (+ 96us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:48:41.632286 (+ 241us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:48:41.632295 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:48:41.633739 (+ 1444us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:48:41.633743 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:48:41.635254 (+ 1511us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:48:41.635269 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:48:41.636877 (+ 1608us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:48:41.636880 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:48:41.637024 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:48:41.637058 (+ 34us) client_negotiation.cc:769] Sending connection context
0925 22:48:41.637971 (+ 913us) client_negotiation.cc:240] Negotiation successful
0925 22:48:41.638044 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":262,"mutex_wait_us":200,"thread_start_us":107,"threads_started":1}
I20260925 22:48:41.641014 29939 tablet_service.cc:1467] Processing CreateTablet for tablet fa7377949ea3455c8c7a245a36051755 (DEFAULT_TABLE table=test-table [id=995109e39c6e4873960ff825fbd7d08b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 22:48:41.641014 29808 tablet_service.cc:1467] Processing CreateTablet for tablet fa7377949ea3455c8c7a245a36051755 (DEFAULT_TABLE table=test-table [id=995109e39c6e4873960ff825fbd7d08b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 22:48:41.642210 29939 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fa7377949ea3455c8c7a245a36051755. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:48:41.642233 29808 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fa7377949ea3455c8c7a245a36051755. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:48:41.643002 30070 tablet_service.cc:1467] Processing CreateTablet for tablet fa7377949ea3455c8c7a245a36051755 (DEFAULT_TABLE table=test-table [id=995109e39c6e4873960ff825fbd7d08b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 22:48:41.644259 30070 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fa7377949ea3455c8c7a245a36051755. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:48:41.649420 30153 tablet_bootstrap.cc:492] T fa7377949ea3455c8c7a245a36051755 P b22bee1e2c3a425985653421554cadfa: Bootstrap starting.
I20260925 22:48:41.651072 30154 tablet_bootstrap.cc:492] T fa7377949ea3455c8c7a245a36051755 P 14fc637d4eca40aabc7cb4d8ff2e1d59: Bootstrap starting.
I20260925 22:48:41.653142 30155 tablet_bootstrap.cc:492] T fa7377949ea3455c8c7a245a36051755 P 98c4f94b242d4f3798a030f1d3b6420b: Bootstrap starting.
I20260925 22:48:41.653771 30154 tablet_bootstrap.cc:654] T fa7377949ea3455c8c7a245a36051755 P 14fc637d4eca40aabc7cb4d8ff2e1d59: Neither blocks nor log segments found. Creating new log.
I20260925 22:48:41.654052 30153 tablet_bootstrap.cc:654] T fa7377949ea3455c8c7a245a36051755 P b22bee1e2c3a425985653421554cadfa: Neither blocks nor log segments found. Creating new log.
I20260925 22:48:41.654807 30154 log.cc:824] T fa7377949ea3455c8c7a245a36051755 P 14fc637d4eca40aabc7cb4d8ff2e1d59: Log is configured to *not* fsync() on all Append() calls
I20260925 22:48:41.654807 30153 log.cc:824] T fa7377949ea3455c8c7a245a36051755 P b22bee1e2c3a425985653421554cadfa: Log is configured to *not* fsync() on all Append() calls
I20260925 22:48:41.655668 30155 tablet_bootstrap.cc:654] T fa7377949ea3455c8c7a245a36051755 P 98c4f94b242d4f3798a030f1d3b6420b: Neither blocks nor log segments found. Creating new log.
I20260925 22:48:41.656688 30155 log.cc:824] T fa7377949ea3455c8c7a245a36051755 P 98c4f94b242d4f3798a030f1d3b6420b: Log is configured to *not* fsync() on all Append() calls
I20260925 22:48:41.657358 30154 tablet_bootstrap.cc:492] T fa7377949ea3455c8c7a245a36051755 P 14fc637d4eca40aabc7cb4d8ff2e1d59: No bootstrap required, opened a new log
I20260925 22:48:41.657598 30154 ts_tablet_manager.cc:1397] T fa7377949ea3455c8c7a245a36051755 P 14fc637d4eca40aabc7cb4d8ff2e1d59: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20260925 22:48:41.657627 30153 tablet_bootstrap.cc:492] T fa7377949ea3455c8c7a245a36051755 P b22bee1e2c3a425985653421554cadfa: No bootstrap required, opened a new log
I20260925 22:48:41.657794 30153 ts_tablet_manager.cc:1397] T fa7377949ea3455c8c7a245a36051755 P b22bee1e2c3a425985653421554cadfa: Time spent bootstrapping tablet: real 0.009s user 0.005s sys 0.000s
I20260925 22:48:41.661929 30155 tablet_bootstrap.cc:492] T fa7377949ea3455c8c7a245a36051755 P 98c4f94b242d4f3798a030f1d3b6420b: No bootstrap required, opened a new log
I20260925 22:48:41.662237 30155 ts_tablet_manager.cc:1397] T fa7377949ea3455c8c7a245a36051755 P 98c4f94b242d4f3798a030f1d3b6420b: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260925 22:48:41.662477 30153 raft_consensus.cc:348] T fa7377949ea3455c8c7a245a36051755 P b22bee1e2c3a425985653421554cadfa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98c4f94b242d4f3798a030f1d3b6420b" member_type: VOTER last_known_addr { host: "127.10.198.3" port: 43691 } } peers { permanent_uuid: "14fc637d4eca40aabc7cb4d8ff2e1d59" member_type: VOTER last_known_addr { host: "127.10.198.1" port: 34281 } } peers { permanent_uuid: "b22bee1e2c3a425985653421554cadfa" member_type: VOTER last_known_addr { host: "127.10.198.2" port: 37035 } }
I20260925 22:48:41.662518 30154 raft_consensus.cc:348] T fa7377949ea3455c8c7a245a36051755 P 14fc637d4eca40aabc7cb4d8ff2e1d59 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98c4f94b242d4f3798a030f1d3b6420b" member_type: VOTER last_known_addr { host: "127.10.198.3" port: 43691 } } peers { permanent_uuid: "14fc637d4eca40aabc7cb4d8ff2e1d59" member_type: VOTER last_known_addr { host: "127.10.198.1" port: 34281 } } peers { permanent_uuid: "b22bee1e2c3a425985653421554cadfa" member_type: VOTER last_known_addr { host: "127.10.198.2" port: 37035 } }
I20260925 22:48:41.662803 30153 raft_consensus.cc:374] T fa7377949ea3455c8c7a245a36051755 P b22bee1e2c3a425985653421554cadfa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:48:41.662803 30154 raft_consensus.cc:374] T fa7377949ea3455c8c7a245a36051755 P 14fc637d4eca40aabc7cb4d8ff2e1d59 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:48:41.662864 30154 raft_consensus.cc:729] T fa7377949ea3455c8c7a245a36051755 P 14fc637d4eca40aabc7cb4d8ff2e1d59 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 14fc637d4eca40aabc7cb4d8ff2e1d59, State: Initialized, Role: FOLLOWER
I20260925 22:48:41.662864 30153 raft_consensus.cc:729] T fa7377949ea3455c8c7a245a36051755 P b22bee1e2c3a425985653421554cadfa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b22bee1e2c3a425985653421554cadfa, State: Initialized, Role: FOLLOWER
I20260925 22:48:41.663384 30153 consensus_queue.cc:260] T fa7377949ea3455c8c7a245a36051755 P b22bee1e2c3a425985653421554cadfa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98c4f94b242d4f3798a030f1d3b6420b" member_type: VOTER last_known_addr { host: "127.10.198.3" port: 43691 } } peers { permanent_uuid: "14fc637d4eca40aabc7cb4d8ff2e1d59" member_type: VOTER last_known_addr { host: "127.10.198.1" port: 34281 } } peers { permanent_uuid: "b22bee1e2c3a425985653421554cadfa" member_type: VOTER last_known_addr { host: "127.10.198.2" port: 37035 } }
I20260925 22:48:41.663384 30154 consensus_queue.cc:260] T fa7377949ea3455c8c7a245a36051755 P 14fc637d4eca40aabc7cb4d8ff2e1d59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98c4f94b242d4f3798a030f1d3b6420b" member_type: VOTER last_known_addr { host: "127.10.198.3" port: 43691 } } peers { permanent_uuid: "14fc637d4eca40aabc7cb4d8ff2e1d59" member_type: VOTER last_known_addr { host: "127.10.198.1" port: 34281 } } peers { permanent_uuid: "b22bee1e2c3a425985653421554cadfa" member_type: VOTER last_known_addr { host: "127.10.198.2" port: 37035 } }
I20260925 22:48:41.664569 30154 ts_tablet_manager.cc:1428] T fa7377949ea3455c8c7a245a36051755 P 14fc637d4eca40aabc7cb4d8ff2e1d59: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260925 22:48:41.664877 30153 ts_tablet_manager.cc:1428] T fa7377949ea3455c8c7a245a36051755 P b22bee1e2c3a425985653421554cadfa: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260925 22:48:41.664968 29874 heartbeater.cc:503] Master 127.10.198.62:33247 was elected leader, sending a full tablet report...
I20260925 22:48:41.665228 30005 heartbeater.cc:503] Master 127.10.198.62:33247 was elected leader, sending a full tablet report...
I20260925 22:48:41.666806 30155 raft_consensus.cc:348] T fa7377949ea3455c8c7a245a36051755 P 98c4f94b242d4f3798a030f1d3b6420b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98c4f94b242d4f3798a030f1d3b6420b" member_type: VOTER last_known_addr { host: "127.10.198.3" port: 43691 } } peers { permanent_uuid: "14fc637d4eca40aabc7cb4d8ff2e1d59" member_type: VOTER last_known_addr { host: "127.10.198.1" port: 34281 } } peers { permanent_uuid: "b22bee1e2c3a425985653421554cadfa" member_type: VOTER last_known_addr { host: "127.10.198.2" port: 37035 } }
I20260925 22:48:41.667122 30155 raft_consensus.cc:374] T fa7377949ea3455c8c7a245a36051755 P 98c4f94b242d4f3798a030f1d3b6420b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:48:41.667191 30155 raft_consensus.cc:729] T fa7377949ea3455c8c7a245a36051755 P 98c4f94b242d4f3798a030f1d3b6420b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98c4f94b242d4f3798a030f1d3b6420b, State: Initialized, Role: FOLLOWER
I20260925 22:48:41.667701 30155 consensus_queue.cc:260] T fa7377949ea3455c8c7a245a36051755 P 98c4f94b242d4f3798a030f1d3b6420b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98c4f94b242d4f3798a030f1d3b6420b" member_type: VOTER last_known_addr { host: "127.10.198.3" port: 43691 } } peers { permanent_uuid: "14fc637d4eca40aabc7cb4d8ff2e1d59" member_type: VOTER last_known_addr { host: "127.10.198.1" port: 34281 } } peers { permanent_uuid: "b22bee1e2c3a425985653421554cadfa" member_type: VOTER last_known_addr { host: "127.10.198.2" port: 37035 } }
I20260925 22:48:41.668808 30136 heartbeater.cc:503] Master 127.10.198.62:33247 was elected leader, sending a full tablet report...
I20260925 22:48:41.668947 30155 ts_tablet_manager.cc:1428] T fa7377949ea3455c8c7a245a36051755 P 98c4f94b242d4f3798a030f1d3b6420b: Time spent starting tablet: real 0.007s user 0.002s sys 0.004s
I20260925 22:48:41.706977 30161 raft_consensus.cc:482] T fa7377949ea3455c8c7a245a36051755 P 98c4f94b242d4f3798a030f1d3b6420b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 22:48:41.707278 30161 raft_consensus.cc:504] T fa7377949ea3455c8c7a245a36051755 P 98c4f94b242d4f3798a030f1d3b6420b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98c4f94b242d4f3798a030f1d3b6420b" member_type: VOTER last_known_addr { host: "127.10.198.3" port: 43691 } } peers { permanent_uuid: "14fc637d4eca40aabc7cb4d8ff2e1d59" member_type: VOTER last_known_addr { host: "127.10.198.1" port: 34281 } } peers { permanent_uuid: "b22bee1e2c3a425985653421554cadfa" member_type: VOTER last_known_addr { host: "127.10.198.2" port: 37035 } }
I20260925 22:48:41.708658 30161 leader_election.cc:288] T fa7377949ea3455c8c7a245a36051755 P 98c4f94b242d4f3798a030f1d3b6420b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 14fc637d4eca40aabc7cb4d8ff2e1d59 (127.10.198.1:34281), b22bee1e2c3a425985653421554cadfa (127.10.198.2:37035)
I20260925 22:48:41.712633 30152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:48:41.709025 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.198.3:35857 (local address 127.10.198.2:37035)
0925 22:48:41.709160 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0925 22:48:41.709165 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:48:41.709188 (+ 23us) server_negotiation.cc:407] Connection header received
0925 22:48:41.709380 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:48:41.709382 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:48:41.709428 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:48:41.709530 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:48:41.709817 (+ 287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:48:41.710331 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:48:41.711541 (+ 1210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:48:41.712028 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:48:41.712156 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:48:41.712213 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0925 22:48:41.712416 (+ 203us) server_negotiation.cc:299] Negotiation successful
0925 22:48:41.712499 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260925 22:48:41.712675 30150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:48:41.708988 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.198.3:33095 (local address 127.10.198.1:34281)
0925 22:48:41.709122 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0925 22:48:41.709128 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 22:48:41.709444 (+ 316us) server_negotiation.cc:407] Connection header received
0925 22:48:41.709588 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:48:41.709590 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:48:41.709633 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:48:41.709717 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:48:41.709985 (+ 268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:48:41.710510 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:48:41.711573 (+ 1063us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:48:41.712051 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:48:41.712180 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:48:41.712248 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0925 22:48:41.712463 (+ 215us) server_negotiation.cc:299] Negotiation successful
0925 22:48:41.712545 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260925 22:48:41.712718 30162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:48:41.708913 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.198.1:34281 (local address 127.10.198.3:33095)
0925 22:48:41.709337 (+ 424us) negotiation.cc:107] Waiting for socket to connect
0925 22:48:41.709352 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 22:48:41.709474 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:48:41.709730 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:48:41.709734 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:48:41.709749 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:48:41.709874 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:48:41.709879 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:48:41.710625 (+ 746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:48:41.710630 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:48:41.711427 (+ 797us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:48:41.711436 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:48:41.712175 (+ 739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:48:41.712178 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:48:41.712339 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:48:41.712357 (+ 18us) client_negotiation.cc:769] Sending connection context
0925 22:48:41.712446 (+ 89us) client_negotiation.cc:240] Negotiation successful
0925 22:48:41.712542 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":346,"mutex_wait_us":64,"thread_start_us":156,"threads_started":1}
I20260925 22:48:41.713110 30138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:48:41.708912 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.198.2:37035 (local address 127.10.198.3:35857)
0925 22:48:41.709120 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0925 22:48:41.709137 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 22:48:41.709259 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:48:41.709545 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:48:41.709548 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:48:41.709566 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:48:41.709694 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:48:41.709701 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:48:41.710455 (+ 754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:48:41.710458 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:48:41.711401 (+ 943us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:48:41.711412 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:48:41.712154 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:48:41.712158 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:48:41.712291 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:48:41.712310 (+ 19us) client_negotiation.cc:769] Sending connection context
0925 22:48:41.712400 (+ 90us) client_negotiation.cc:240] Negotiation successful
0925 22:48:41.712956 (+ 556us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120,"mutex_wait_us":417}
I20260925 22:48:41.713608 29828 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa7377949ea3455c8c7a245a36051755" candidate_uuid: "98c4f94b242d4f3798a030f1d3b6420b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "14fc637d4eca40aabc7cb4d8ff2e1d59" is_pre_election: true
I20260925 22:48:41.713842 29959 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa7377949ea3455c8c7a245a36051755" candidate_uuid: "98c4f94b242d4f3798a030f1d3b6420b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b22bee1e2c3a425985653421554cadfa" is_pre_election: true
I20260925 22:48:41.713984 29828 raft_consensus.cc:2454] T fa7377949ea3455c8c7a245a36051755 P 14fc637d4eca40aabc7cb4d8ff2e1d59 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 98c4f94b242d4f3798a030f1d3b6420b in term 0.
I20260925 22:48:41.714097 29959 raft_consensus.cc:2454] T fa7377949ea3455c8c7a245a36051755 P b22bee1e2c3a425985653421554cadfa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 98c4f94b242d4f3798a030f1d3b6420b in term 0.
I20260925 22:48:41.714571 30026 leader_election.cc:302] T fa7377949ea3455c8c7a245a36051755 P 98c4f94b242d4f3798a030f1d3b6420b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 14fc637d4eca40aabc7cb4d8ff2e1d59, 98c4f94b242d4f3798a030f1d3b6420b, b22bee1e2c3a425985653421554cadfa; no voters:
I20260925 22:48:41.714924 30161 raft_consensus.cc:2781] T fa7377949ea3455c8c7a245a36051755 P 98c4f94b242d4f3798a030f1d3b6420b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 22:48:41.715034 30161 raft_consensus.cc:482] T fa7377949ea3455c8c7a245a36051755 P 98c4f94b242d4f3798a030f1d3b6420b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 22:48:41.715095 30161 raft_consensus.cc:3037] T fa7377949ea3455c8c7a245a36051755 P 98c4f94b242d4f3798a030f1d3b6420b [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:48:41.716349 30161 raft_consensus.cc:504] T fa7377949ea3455c8c7a245a36051755 P 98c4f94b242d4f3798a030f1d3b6420b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98c4f94b242d4f3798a030f1d3b6420b" member_type: VOTER last_known_addr { host: "127.10.198.3" port: 43691 } } peers { permanent_uuid: "14fc637d4eca40aabc7cb4d8ff2e1d59" member_type: VOTER last_known_addr { host: "127.10.198.1" port: 34281 } } peers { permanent_uuid: "b22bee1e2c3a425985653421554cadfa" member_type: VOTER last_known_addr { host: "127.10.198.2" port: 37035 } }
I20260925 22:48:41.716758 30161 leader_election.cc:288] T fa7377949ea3455c8c7a245a36051755 P 98c4f94b242d4f3798a030f1d3b6420b [CANDIDATE]: Term 1 election: Requested vote from peers 14fc637d4eca40aabc7cb4d8ff2e1d59 (127.10.198.1:34281), b22bee1e2c3a425985653421554cadfa (127.10.198.2:37035)
I20260925 22:48:41.717389 29828 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa7377949ea3455c8c7a245a36051755" candidate_uuid: "98c4f94b242d4f3798a030f1d3b6420b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "14fc637d4eca40aabc7cb4d8ff2e1d59"
I20260925 22:48:41.717581 29828 raft_consensus.cc:3037] T fa7377949ea3455c8c7a245a36051755 P 14fc637d4eca40aabc7cb4d8ff2e1d59 [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:48:41.717590 29959 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa7377949ea3455c8c7a245a36051755" candidate_uuid: "98c4f94b242d4f3798a030f1d3b6420b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b22bee1e2c3a425985653421554cadfa"
I20260925 22:48:41.717738 29959 raft_consensus.cc:3037] T fa7377949ea3455c8c7a245a36051755 P b22bee1e2c3a425985653421554cadfa [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:48:41.718806 29828 raft_consensus.cc:2454] T fa7377949ea3455c8c7a245a36051755 P 14fc637d4eca40aabc7cb4d8ff2e1d59 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 98c4f94b242d4f3798a030f1d3b6420b in term 1.
I20260925 22:48:41.718819 29959 raft_consensus.cc:2454] T fa7377949ea3455c8c7a245a36051755 P b22bee1e2c3a425985653421554cadfa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 98c4f94b242d4f3798a030f1d3b6420b in term 1.
I20260925 22:48:41.719252 30026 leader_election.cc:302] T fa7377949ea3455c8c7a245a36051755 P 98c4f94b242d4f3798a030f1d3b6420b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 14fc637d4eca40aabc7cb4d8ff2e1d59, 98c4f94b242d4f3798a030f1d3b6420b, b22bee1e2c3a425985653421554cadfa; no voters:
I20260925 22:48:41.719467 30161 raft_consensus.cc:2781] T fa7377949ea3455c8c7a245a36051755 P 98c4f94b242d4f3798a030f1d3b6420b [term 1 FOLLOWER]: Leader election won for term 1
I20260925 22:48:41.719734 30161 raft_consensus.cc:686] T fa7377949ea3455c8c7a245a36051755 P 98c4f94b242d4f3798a030f1d3b6420b [term 1 LEADER]: Becoming Leader. State: Replica: 98c4f94b242d4f3798a030f1d3b6420b, State: Running, Role: LEADER
I20260925 22:48:41.720129 30161 consensus_queue.cc:237] T fa7377949ea3455c8c7a245a36051755 P 98c4f94b242d4f3798a030f1d3b6420b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98c4f94b242d4f3798a030f1d3b6420b" member_type: VOTER last_known_addr { host: "127.10.198.3" port: 43691 } } peers { permanent_uuid: "14fc637d4eca40aabc7cb4d8ff2e1d59" member_type: VOTER last_known_addr { host: "127.10.198.1" port: 34281 } } peers { permanent_uuid: "b22bee1e2c3a425985653421554cadfa" member_type: VOTER last_known_addr { host: "127.10.198.2" port: 37035 } }
I20260925 22:48:41.723733 29685 catalog_manager.cc:5495] T fa7377949ea3455c8c7a245a36051755 P 98c4f94b242d4f3798a030f1d3b6420b reported cstate change: term changed from 0 to 1, leader changed from <none> to 98c4f94b242d4f3798a030f1d3b6420b (127.10.198.3). New cstate: current_term: 1 leader_uuid: "98c4f94b242d4f3798a030f1d3b6420b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98c4f94b242d4f3798a030f1d3b6420b" member_type: VOTER last_known_addr { host: "127.10.198.3" port: 43691 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "14fc637d4eca40aabc7cb4d8ff2e1d59" member_type: VOTER last_known_addr { host: "127.10.198.1" port: 34281 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b22bee1e2c3a425985653421554cadfa" member_type: VOTER last_known_addr { host: "127.10.198.2" port: 37035 } health_report { overall_health: UNKNOWN } } }
I20260925 22:48:41.742355 30149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:48:41.738707 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35708 (local address 127.10.198.3:43691)
0925 22:48:41.738849 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0925 22:48:41.738853 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:48:41.738888 (+ 35us) server_negotiation.cc:407] Connection header received
0925 22:48:41.738965 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:48:41.738967 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:48:41.739026 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:48:41.739114 (+ 88us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 22:48:41.739349 (+ 235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:48:41.739875 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:48:41.740772 (+ 897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:48:41.740997 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:48:41.741087 (+ 90us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:48:41.741378 (+ 291us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 22:48:41.741813 (+ 435us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 22:48:41.741928 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0925 22:48:41.742054 (+ 126us) server_negotiation.cc:299] Negotiation successful
0925 22:48:41.742127 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260925 22:48:41.749209 29959 raft_consensus.cc:1261] T fa7377949ea3455c8c7a245a36051755 P b22bee1e2c3a425985653421554cadfa [term 1 FOLLOWER]: Refusing update from remote peer 98c4f94b242d4f3798a030f1d3b6420b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 22:48:41.749209 29828 raft_consensus.cc:1261] T fa7377949ea3455c8c7a245a36051755 P 14fc637d4eca40aabc7cb4d8ff2e1d59 [term 1 FOLLOWER]: Refusing update from remote peer 98c4f94b242d4f3798a030f1d3b6420b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 22:48:41.750306 30163 consensus_queue.cc:1035] T fa7377949ea3455c8c7a245a36051755 P 98c4f94b242d4f3798a030f1d3b6420b [LEADER]: Connected to new peer: Peer: permanent_uuid: "b22bee1e2c3a425985653421554cadfa" member_type: VOTER last_known_addr { host: "127.10.198.2" port: 37035 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 22:48:41.750579 30161 consensus_queue.cc:1035] T fa7377949ea3455c8c7a245a36051755 P 98c4f94b242d4f3798a030f1d3b6420b [LEADER]: Connected to new peer: Peer: permanent_uuid: "14fc637d4eca40aabc7cb4d8ff2e1d59" member_type: VOTER last_known_addr { host: "127.10.198.1" port: 34281 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 22:48:41.760334 30167 mvcc.cc:205] Tried to move back new op lower bound from 7333382233075290112 to 7333382232968609792. Current Snapshot: MvccSnapshot[applied={T|T < 7333382233075290112}]
I20260925 22:48:41.772516 29685 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:38196:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 22:48:41.772737 29685 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:38196:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 22:48:41.776274 29685 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ed0963ae2e61483bbe3719bab08b719f: Sending DeleteTablet for 3 replicas of tablet fa7377949ea3455c8c7a245a36051755
I20260925 22:48:41.777657 29808 tablet_service.cc:1514] Processing DeleteTablet for tablet fa7377949ea3455c8c7a245a36051755 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 22:48:41 UTC) from {username='slave'} at 127.0.0.1:58564
I20260925 22:48:41.778316 30176 tablet_replica.cc:330] T fa7377949ea3455c8c7a245a36051755 P 14fc637d4eca40aabc7cb4d8ff2e1d59: stopping tablet replica
I20260925 22:48:41.778524 30070 tablet_service.cc:1514] Processing DeleteTablet for tablet fa7377949ea3455c8c7a245a36051755 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 22:48:41 UTC) from {username='slave'} at 127.0.0.1:35706
I20260925 22:48:41.778966 30176 raft_consensus.cc:2229] T fa7377949ea3455c8c7a245a36051755 P 14fc637d4eca40aabc7cb4d8ff2e1d59 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 22:48:41.779080 29939 tablet_service.cc:1514] Processing DeleteTablet for tablet fa7377949ea3455c8c7a245a36051755 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 22:48:41 UTC) from {username='slave'} at 127.0.0.1:43606
I20260925 22:48:41.779444 30176 raft_consensus.cc:2258] T fa7377949ea3455c8c7a245a36051755 P 14fc637d4eca40aabc7cb4d8ff2e1d59 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 22:48:41.780078 30177 tablet_replica.cc:330] T fa7377949ea3455c8c7a245a36051755 P 98c4f94b242d4f3798a030f1d3b6420b: stopping tablet replica
I20260925 22:48:41.780444 30177 raft_consensus.cc:2229] T fa7377949ea3455c8c7a245a36051755 P 98c4f94b242d4f3798a030f1d3b6420b [term 1 LEADER]: Raft consensus shutting down.
I20260925 22:48:41.780859 11032 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrWXm8X/build/debug/bin/kudu with pid 29746
I20260925 22:48:41.782209 30178 tablet_replica.cc:330] T fa7377949ea3455c8c7a245a36051755 P b22bee1e2c3a425985653421554cadfa: stopping tablet replica
I20260925 22:48:41.782608 30178 raft_consensus.cc:2229] T fa7377949ea3455c8c7a245a36051755 P b22bee1e2c3a425985653421554cadfa [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 22:48:41.782994 30178 raft_consensus.cc:2258] T fa7377949ea3455c8c7a245a36051755 P b22bee1e2c3a425985653421554cadfa [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 22:48:41.783412 30177 raft_consensus.cc:2258] T fa7377949ea3455c8c7a245a36051755 P 98c4f94b242d4f3798a030f1d3b6420b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 22:48:41.785214 30178 ts_tablet_manager.cc:1905] T fa7377949ea3455c8c7a245a36051755 P b22bee1e2c3a425985653421554cadfa: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 22:48:41.785323 30177 ts_tablet_manager.cc:1905] T fa7377949ea3455c8c7a245a36051755 P 98c4f94b242d4f3798a030f1d3b6420b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 22:48:41.788336 30177 ts_tablet_manager.cc:1918] T fa7377949ea3455c8c7a245a36051755 P 98c4f94b242d4f3798a030f1d3b6420b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 22:48:41.788336 30178 ts_tablet_manager.cc:1918] T fa7377949ea3455c8c7a245a36051755 P b22bee1e2c3a425985653421554cadfa: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 22:48:41.788481 30177 log.cc:1196] T fa7377949ea3455c8c7a245a36051755 P 98c4f94b242d4f3798a030f1d3b6420b: Deleting WAL directory at /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790376259650642-11032-0/minicluster-data/ts-2/wal/wals/fa7377949ea3455c8c7a245a36051755
I20260925 22:48:41.788481 30178 log.cc:1196] T fa7377949ea3455c8c7a245a36051755 P b22bee1e2c3a425985653421554cadfa: Deleting WAL directory at /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790376259650642-11032-0/minicluster-data/ts-1/wal/wals/fa7377949ea3455c8c7a245a36051755
I20260925 22:48:41.788828 30177 ts_tablet_manager.cc:1939] T fa7377949ea3455c8c7a245a36051755 P 98c4f94b242d4f3798a030f1d3b6420b: Deleting consensus metadata
I20260925 22:48:41.788828 30178 ts_tablet_manager.cc:1939] T fa7377949ea3455c8c7a245a36051755 P b22bee1e2c3a425985653421554cadfa: Deleting consensus metadata
I20260925 22:48:41.790293 29670 catalog_manager.cc:4841] TS b22bee1e2c3a425985653421554cadfa (127.10.198.2:37035): tablet fa7377949ea3455c8c7a245a36051755 (table test-table [id=995109e39c6e4873960ff825fbd7d08b]) successfully deleted
W20260925 22:48:41.790395 29673 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.10.198.1:34281 (error 108)
I20260925 22:48:41.790295 29671 catalog_manager.cc:4841] TS 98c4f94b242d4f3798a030f1d3b6420b (127.10.198.3:43691): tablet fa7377949ea3455c8c7a245a36051755 (table test-table [id=995109e39c6e4873960ff825fbd7d08b]) successfully deleted
I20260925 22:48:41.791546 11032 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrWXm8X/build/debug/bin/kudu with pid 29877
I20260925 22:48:41.791656 30148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:48:41.791268 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.198.1:34281 (local address 127.0.0.1:58576)
0925 22:48:41.791472 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0925 22:48:41.791550 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.198.1:34281: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120}
W20260925 22:48:41.792333 29673 catalog_manager.cc:4584] TS 14fc637d4eca40aabc7cb4d8ff2e1d59 (127.10.198.1:34281): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet fa7377949ea3455c8c7a245a36051755: Network error: Client connection negotiation failed: client connection to 127.10.198.1:34281: connect: Connection refused (error 111)
I20260925 22:48:41.798986 11032 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrWXm8X/build/debug/bin/kudu with pid 30008
I20260925 22:48:41.805198 11032 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrWXm8X/build/debug/bin/kudu with pid 29655
2026-09-25T22:48:41Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (1146 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790376259650642-11032-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 22:48:41 dist-test-slave-5vdr krb5kdc[30182](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 22:48:41 dist-test-slave-5vdr krb5kdc[30182](info): set up 2 sockets
Sep 25 22:48:41 dist-test-slave-5vdr krb5kdc[30182](info): commencing operation
krb5kdc: starting...
W20260925 22:48:42.855976 11032 mini_kdc.cc:121] Time spent starting KDC: real 1.028s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790376259650642-11032-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790376259650642-11032-0/krb5kdc/test-user.keytab.
Sep 25 22:48:42 dist-test-slave-5vdr krb5kdc[30182](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376522, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T22:48:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T22:48:42Z Disabled control of system clock
WARNING: no policy specified for kudu/127.10.198.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.198.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.198.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790376259650642-11032-0/krb5kdc/kudu_127.10.198.62.keytab.
Entry for principal kudu/127.10.198.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790376259650642-11032-0/krb5kdc/kudu_127.10.198.62.keytab.
I20260925 22:48:42.986955 11032 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrWXm8X/build/debug/bin/kudu
/tmp/dist-test-taskrWXm8X/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790376259650642-11032-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790376259650642-11032-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790376259650642-11032-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790376259650642-11032-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.198.62:38175
--webserver_interface=127.10.198.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790376259650642-11032-0/krb5kdc/kudu_127.10.198.62.keytab
--principal=kudu/127.10.198.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.198.20:41677
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.198.62:38175
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790376259650642-11032-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790376259650642-11032-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790376259650642-11032-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:48:43.104149 30195 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:48:43.104491 30195 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:48:43.104537 30195 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:48:43.108927 30195 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 22:48:43.109014 30195 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:48:43.109040 30195 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:48:43.109061 30195 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 22:48:43.109077 30195 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 22:48:43.114861 30195 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.198.20:41677
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790376259650642-11032-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790376259650642-11032-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.198.62:38175
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790376259650642-11032-0/krb5kdc/kudu_127.10.198.62.keytab
--principal=kudu/127.10.198.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.198.62:38175
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790376259650642-11032-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.198.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30195
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790376259650642-11032-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:09:16 UTC on ccycloud.quasar-afmhyw.root.comops.site
build id 4673379
I20260925 22:48:43.116209 30195 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:48:43.117350 30195 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:48:43.127400 30200 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:48:43.127404 30203 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:48:43.127400 30201 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:48:43.127898 30195 server_base.cc:1034] running on GCE node
I20260925 22:48:43.128577 30195 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:48:43.129892 30195 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:48:43.131117 30195 hybrid_clock.cc:648] HybridClock initialized: now 1790376523131101 us; error 40 us; skew 500 ppm
Sep 25 22:48:43 dist-test-slave-5vdr krb5kdc[30182](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376523, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.198.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:48:43.134368 30195 init.cc:377] Logged in from keytab as kudu/127.10.198.62@KRBTEST.COM (short username kudu)
I20260925 22:48:43.135406 30195 webserver.cc:460] Webserver started at http://127.10.198.62:45481/ using document root <none> and password file <none>
I20260925 22:48:43.136085 30195 fs_manager.cc:359] Metadata directory not provided
I20260925 22:48:43.136161 30195 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:48:43.136358 30195 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:48:43.138600 30195 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790376259650642-11032-0/minicluster-data/master-0/data/instance:
uuid: "c25248a58bee4ebcaaab275864d564da"
format_stamp: "Formatted at 2026-09-25 22:48:43 on dist-test-slave-5vdr"
server_key: "c37e845a2db97d82314ae4efc5006cbc"
server_key_iv: "774f4ec48685f4b486adf1fbe6ed9f97"
server_key_version: "encryptionkey@0"
I20260925 22:48:43.139178 30195 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790376259650642-11032-0/minicluster-data/master-0/wal/instance:
uuid: "c25248a58bee4ebcaaab275864d564da"
format_stamp: "Formatted at 2026-09-25 22:48:43 on dist-test-slave-5vdr"
server_key: "c37e845a2db97d82314ae4efc5006cbc"
server_key_iv: "774f4ec48685f4b486adf1fbe6ed9f97"
server_key_version: "encryptionkey@0"
I20260925 22:48:43.143527 30195 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260925 22:48:43.146520 30210 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:48:43.148321 30195 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 22:48:43.148458 30195 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790376259650642-11032-0/minicluster-data/master-0/data,/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790376259650642-11032-0/minicluster-data/master-0/wal
uuid: "c25248a58bee4ebcaaab275864d564da"
format_stamp: "Formatted at 2026-09-25 22:48:43 on dist-test-slave-5vdr"
server_key: "c37e845a2db97d82314ae4efc5006cbc"
server_key_iv: "774f4ec48685f4b486adf1fbe6ed9f97"
server_key_version: "encryptionkey@0"
I20260925 22:48:43.148587 30195 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790376259650642-11032-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790376259650642-11032-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790376259650642-11032-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:48:43.164222 30195 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:48:43.167605 30195 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:48:43.167796 30195 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:48:43.175655 30195 rpc_server.cc:307] RPC server started. Bound to: 127.10.198.62:38175
I20260925 22:48:43.175669 30262 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.198.62:38175 every 8 connection(s)
I20260925 22:48:43.176862 30195 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790376259650642-11032-0/minicluster-data/master-0/data/info.pb
I20260925 22:48:43.180117 30263 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:48:43.184366 11032 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrWXm8X/build/debug/bin/kudu as pid 30195
I20260925 22:48:43.184530 11032 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790376259650642-11032-0/minicluster-data/master-0/wal/instance
I20260925 22:48:43.184846 11032 external_mini_cluster.cc:1428] Setting key e954ae70079357a81b60cec5ef2a4696
I20260925 22:48:43.187124 30263 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c25248a58bee4ebcaaab275864d564da: Bootstrap starting.
I20260925 22:48:43.190083 30263 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c25248a58bee4ebcaaab275864d564da: Neither blocks nor log segments found. Creating new log.
I20260925 22:48:43.191026 30263 log.cc:824] T 00000000000000000000000000000000 P c25248a58bee4ebcaaab275864d564da: Log is configured to *not* fsync() on all Append() calls
I20260925 22:48:43.193782 30263 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c25248a58bee4ebcaaab275864d564da: No bootstrap required, opened a new log
Sep 25 22:48:43 dist-test-slave-5vdr krb5kdc[30182](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376522, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.198.62@KRBTEST.COM
I20260925 22:48:43.197825 30263 raft_consensus.cc:348] T 00000000000000000000000000000000 P c25248a58bee4ebcaaab275864d564da [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c25248a58bee4ebcaaab275864d564da" member_type: VOTER last_known_addr { host: "127.10.198.62" port: 38175 } }
I20260925 22:48:43.198145 30263 raft_consensus.cc:374] T 00000000000000000000000000000000 P c25248a58bee4ebcaaab275864d564da [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:48:43.198238 30263 raft_consensus.cc:729] T 00000000000000000000000000000000 P c25248a58bee4ebcaaab275864d564da [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c25248a58bee4ebcaaab275864d564da, State: Initialized, Role: FOLLOWER
I20260925 22:48:43.198786 30263 consensus_queue.cc:260] T 00000000000000000000000000000000 P c25248a58bee4ebcaaab275864d564da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c25248a58bee4ebcaaab275864d564da" member_type: VOTER last_known_addr { host: "127.10.198.62" port: 38175 } }
I20260925 22:48:43.198959 30263 raft_consensus.cc:388] T 00000000000000000000000000000000 P c25248a58bee4ebcaaab275864d564da [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 22:48:43.199051 30263 raft_consensus.cc:482] T 00000000000000000000000000000000 P c25248a58bee4ebcaaab275864d564da [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 22:48:43.199168 30263 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c25248a58bee4ebcaaab275864d564da [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:48:43.200300 30263 raft_consensus.cc:504] T 00000000000000000000000000000000 P c25248a58bee4ebcaaab275864d564da [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c25248a58bee4ebcaaab275864d564da" member_type: VOTER last_known_addr { host: "127.10.198.62" port: 38175 } }
I20260925 22:48:43.200747 30263 leader_election.cc:302] T 00000000000000000000000000000000 P c25248a58bee4ebcaaab275864d564da [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c25248a58bee4ebcaaab275864d564da; no voters:
I20260925 22:48:43.201169 30263 leader_election.cc:288] T 00000000000000000000000000000000 P c25248a58bee4ebcaaab275864d564da [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 22:48:43.201267 30268 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c25248a58bee4ebcaaab275864d564da [term 1 FOLLOWER]: Leader election won for term 1
I20260925 22:48:43.201485 30268 raft_consensus.cc:686] T 00000000000000000000000000000000 P c25248a58bee4ebcaaab275864d564da [term 1 LEADER]: Becoming Leader. State: Replica: c25248a58bee4ebcaaab275864d564da, State: Running, Role: LEADER
I20260925 22:48:43.201822 30268 consensus_queue.cc:237] T 00000000000000000000000000000000 P c25248a58bee4ebcaaab275864d564da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c25248a58bee4ebcaaab275864d564da" member_type: VOTER last_known_addr { host: "127.10.198.62" port: 38175 } }
I20260925 22:48:43.202152 30266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:48:43.186459 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55318 (local address 127.10.198.62:38175)
0925 22:48:43.186951 (+ 492us) server_negotiation.cc:206] Beginning negotiation
0925 22:48:43.186968 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0925 22:48:43.187009 (+ 41us) server_negotiation.cc:407] Connection header received
0925 22:48:43.187802 (+ 793us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:48:43.187823 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:48:43.188300 (+ 477us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:48:43.188626 (+ 326us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:48:43.189640 (+ 1014us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:48:43.190811 (+ 1171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:48:43.192034 (+ 1223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:48:43.192559 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:48:43.192701 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:48:43.195713 (+ 3012us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:48:43.195720 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:48:43.195740 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:48:43.195776 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:48:43.198351 (+ 2575us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:48:43.198909 (+ 558us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:48:43.198922 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:48:43.198928 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:48:43.199119 (+ 191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:48:43.199451 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:48:43.199455 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:48:43.199456 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:48:43.200005 (+ 549us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:48:43.200254 (+ 249us) server_negotiation.cc:1035] Waiting for connection context
0925 22:48:43.200552 (+ 298us) server_negotiation.cc:299] Negotiation successful
0925 22:48:43.200926 (+ 374us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":324,"thread_start_us":167,"threads_started":1}
I20260925 22:48:43.202605 30263 sys_catalog.cc:564] T 00000000000000000000000000000000 P c25248a58bee4ebcaaab275864d564da [sys.catalog]: configured and running, proceeding with master startup.
I20260925 22:48:43.203085 30268 sys_catalog.cc:455] T 00000000000000000000000000000000 P c25248a58bee4ebcaaab275864d564da [sys.catalog]: SysCatalogTable state changed. Reason: New leader c25248a58bee4ebcaaab275864d564da. Latest consensus state: current_term: 1 leader_uuid: "c25248a58bee4ebcaaab275864d564da" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c25248a58bee4ebcaaab275864d564da" member_type: VOTER last_known_addr { host: "127.10.198.62" port: 38175 } } }
I20260925 22:48:43.203226 30268 sys_catalog.cc:458] T 00000000000000000000000000000000 P c25248a58bee4ebcaaab275864d564da [sys.catalog]: This master's current role is: LEADER
I20260925 22:48:43.203670 30269 sys_catalog.cc:455] T 00000000000000000000000000000000 P c25248a58bee4ebcaaab275864d564da [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c25248a58bee4ebcaaab275864d564da" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c25248a58bee4ebcaaab275864d564da" member_type: VOTER last_known_addr { host: "127.10.198.62" port: 38175 } } }
I20260925 22:48:43.203895 30269 sys_catalog.cc:458] T 00000000000000000000000000000000 P c25248a58bee4ebcaaab275864d564da [sys.catalog]: This master's current role is: LEADER
W20260925 22:48:43.208030 30283 catalog_manager.cc:1558] T 00000000000000000000000000000000 P c25248a58bee4ebcaaab275864d564da: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 22:48:43.208128 30283 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 22:48:43.208212 30272 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 22:48:43.209319 30272 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 22:48:43.215453 30272 catalog_manager.cc:1347] Generated new cluster ID: 5d7bc9410efc400b9eaab5e0cf089f97
I20260925 22:48:43.215553 30272 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 22:48:43.222469 30272 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 22:48:43.223547 30272 catalog_manager.cc:1504] Loading token signing keys...
I20260925 22:48:43.233428 30272 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c25248a58bee4ebcaaab275864d564da: Generated new TSK 0
I20260925 22:48:43.234447 30272 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.10.198.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.198.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.198.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790376259650642-11032-0/krb5kdc/kudu_127.10.198.1.keytab.
Entry for principal kudu/127.10.198.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790376259650642-11032-0/krb5kdc/kudu_127.10.198.1.keytab.
I20260925 22:48:43.267557 11032 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrWXm8X/build/debug/bin/kudu
/tmp/dist-test-taskrWXm8X/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790376259650642-11032-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790376259650642-11032-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790376259650642-11032-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790376259650642-11032-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.198.1:0
--local_ip_for_outbound_sockets=127.10.198.1
--webserver_interface=127.10.198.1
--webserver_port=0
--tserver_master_addrs=127.10.198.62:38175
--keytab_file=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790376259650642-11032-0/krb5kdc/kudu_127.10.198.1.keytab
--principal=kudu/127.10.198.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.198.20:41677
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790376259650642-11032-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790376259650642-11032-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790376259650642-11032-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:48:43.391017 30289 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:48:43.391377 30289 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:48:43.391476 30289 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:48:43.395821 30289 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:48:43.395994 30289 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:48:43.396090 30289 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.198.1
I20260925 22:48:43.401973 30289 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.198.20:41677
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790376259650642-11032-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790376259650642-11032-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790376259650642-11032-0/krb5kdc/kudu_127.10.198.1.keytab
--principal=kudu/127.10.198.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.198.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790376259650642-11032-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.198.1
--webserver_port=0
--tserver_master_addrs=127.10.198.62:38175
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30289
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.198.1
--log_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790376259650642-11032-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:09:16 UTC on ccycloud.quasar-afmhyw.root.comops.site
build id 4673379
I20260925 22:48:43.403277 30289 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:48:43.404578 30289 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:48:43.415089 30297 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:48:43.415089 30295 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:48:43.415103 30294 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:48:43.415361 30289 server_base.cc:1034] running on GCE node
I20260925 22:48:43.416090 30289 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:48:43.416927 30289 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:48:43.418112 30289 hybrid_clock.cc:648] HybridClock initialized: now 1790376523418100 us; error 33 us; skew 500 ppm
Sep 25 22:48:43 dist-test-slave-5vdr krb5kdc[30182](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376523, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.198.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:48:43.421535 30289 init.cc:377] Logged in from keytab as kudu/127.10.198.1@KRBTEST.COM (short username kudu)
I20260925 22:48:43.422600 30289 webserver.cc:460] Webserver started at http://127.10.198.1:46607/ using document root <none> and password file <none>
I20260925 22:48:43.423254 30289 fs_manager.cc:359] Metadata directory not provided
I20260925 22:48:43.423367 30289 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:48:43.423624 30289 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:48:43.426100 30289 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790376259650642-11032-0/minicluster-data/ts-0/data/instance:
uuid: "cd5b689f63d04cbebf9b88202711232d"
format_stamp: "Formatted at 2026-09-25 22:48:43 on dist-test-slave-5vdr"
server_key: "61ba3b03e543654d3f234eb76f9196ff"
server_key_iv: "1165a2348321c14a1a7f3d588705eb4a"
server_key_version: "encryptionkey@0"
I20260925 22:48:43.426844 30289 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790376259650642-11032-0/minicluster-data/ts-0/wal/instance:
uuid: "cd5b689f63d04cbebf9b88202711232d"
format_stamp: "Formatted at 2026-09-25 22:48:43 on dist-test-slave-5vdr"
server_key: "61ba3b03e543654d3f234eb76f9196ff"
server_key_iv: "1165a2348321c14a1a7f3d588705eb4a"
server_key_version: "encryptionkey@0"
I20260925 22:48:43.431273 30289 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.005s
I20260925 22:48:43.434228 30304 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:48:43.435627 30289 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260925 22:48:43.435813 30289 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790376259650642-11032-0/minicluster-data/ts-0/data,/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790376259650642-11032-0/minicluster-data/ts-0/wal
uuid: "cd5b689f63d04cbebf9b88202711232d"
format_stamp: "Formatted at 2026-09-25 22:48:43 on dist-test-slave-5vdr"
server_key: "61ba3b03e543654d3f234eb76f9196ff"
server_key_iv: "1165a2348321c14a1a7f3d588705eb4a"
server_key_version: "encryptionkey@0"
I20260925 22:48:43.436030 30289 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790376259650642-11032-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790376259650642-11032-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790376259650642-11032-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:48:43.449277 30289 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:48:43.452915 30289 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:48:43.453161 30289 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:48:43.453904 30289 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 22:48:43.454849 30289 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:48:43.454931 30289 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:48:43.455005 30289 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:48:43.455058 30289 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 22:48:43.465315 30289 rpc_server.cc:307] RPC server started. Bound to: 127.10.198.1:40675
I20260925 22:48:43.465333 30417 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.198.1:40675 every 8 connection(s)
I20260925 22:48:43.466480 30289 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790376259650642-11032-0/minicluster-data/ts-0/data/info.pb
I20260925 22:48:43.475631 11032 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrWXm8X/build/debug/bin/kudu as pid 30289
I20260925 22:48:43.475821 11032 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790376259650642-11032-0/minicluster-data/ts-0/wal/instance
I20260925 22:48:43.476156 11032 external_mini_cluster.cc:1428] Setting key 4b901129cf694f671509649d45bbbcd5
Sep 25 22:48:43 dist-test-slave-5vdr krb5kdc[30182](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376523, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.198.1@KRBTEST.COM for kudu/127.10.198.62@KRBTEST.COM
I20260925 22:48:43.482380 30266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:48:43.468619 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.198.1:36133 (local address 127.10.198.62:38175)
0925 22:48:43.468784 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0925 22:48:43.468788 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:48:43.470027 (+ 1239us) server_negotiation.cc:407] Connection header received
0925 22:48:43.470947 (+ 920us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:48:43.470950 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:48:43.471002 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:48:43.471086 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:48:43.472528 (+ 1442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:48:43.473071 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:48:43.474315 (+ 1244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:48:43.474544 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:48:43.474635 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:48:43.478033 (+ 3398us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:48:43.478037 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:48:43.478039 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:48:43.478065 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:48:43.479678 (+ 1613us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:48:43.480386 (+ 708us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:48:43.480392 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:48:43.480399 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:48:43.480483 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:48:43.480907 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:48:43.480911 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:48:43.480912 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:48:43.481127 (+ 215us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:48:43.481261 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0925 22:48:43.482071 (+ 810us) server_negotiation.cc:299] Negotiation successful
0925 22:48:43.482239 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260925 22:48:43.483704 30420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:48:43.469070 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.198.62:38175 (local address 127.10.198.1:36133)
0925 22:48:43.469839 (+ 769us) negotiation.cc:107] Waiting for socket to connect
0925 22:48:43.469891 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0925 22:48:43.470739 (+ 848us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:48:43.471279 (+ 540us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:48:43.471287 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:48:43.471704 (+ 417us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:48:43.472359 (+ 655us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:48:43.472375 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:48:43.473208 (+ 833us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:48:43.473210 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:48:43.474170 (+ 960us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:48:43.474182 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:48:43.474754 (+ 572us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:48:43.474758 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:48:43.474940 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:48:43.475752 (+ 812us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:48:43.475806 (+ 54us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:48:43.477824 (+ 2018us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:48:43.479892 (+ 2068us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:48:43.479901 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:48:43.479923 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:48:43.480255 (+ 332us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:48:43.480595 (+ 340us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:48:43.480598 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:48:43.480600 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:48:43.480785 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:48:43.481256 (+ 471us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:48:43.481264 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:48:43.481666 (+ 402us) client_negotiation.cc:769] Sending connection context
0925 22:48:43.481942 (+ 276us) client_negotiation.cc:240] Negotiation successful
0925 22:48:43.482377 (+ 435us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":544,"thread_start_us":143,"threads_started":1}
I20260925 22:48:43.485388 30418 heartbeater.cc:347] Connected to a master server at 127.10.198.62:38175
I20260925 22:48:43.485711 30418 heartbeater.cc:464] Registering TS with master...
I20260925 22:48:43.486315 30418 heartbeater.cc:511] Master 127.10.198.62:38175 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.10.198.2@KRBTEST.COM; defaulting to no policy
I20260925 22:48:43.488270 30227 ts_manager.cc:194] Registered new tserver with Master: cd5b689f63d04cbebf9b88202711232d (127.10.198.1:40675)
I20260925 22:48:43.489500 30227 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.198.1@KRBTEST.COM'} at 127.10.198.1:36133
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.198.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.198.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790376259650642-11032-0/krb5kdc/kudu_127.10.198.2.keytab.
Entry for principal kudu/127.10.198.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790376259650642-11032-0/krb5kdc/kudu_127.10.198.2.keytab.
I20260925 22:48:43.507011 11032 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrWXm8X/build/debug/bin/kudu
/tmp/dist-test-taskrWXm8X/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790376259650642-11032-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790376259650642-11032-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790376259650642-11032-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790376259650642-11032-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.198.2:0
--local_ip_for_outbound_sockets=127.10.198.2
--webserver_interface=127.10.198.2
--webserver_port=0
--tserver_master_addrs=127.10.198.62:38175
--keytab_file=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790376259650642-11032-0/krb5kdc/kudu_127.10.198.2.keytab
--principal=kudu/127.10.198.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.198.20:41677
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790376259650642-11032-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790376259650642-11032-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790376259650642-11032-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:48:43.630903 30423 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:48:43.631245 30423 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:48:43.631294 30423 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:48:43.636224 30423 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:48:43.636314 30423 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:48:43.636423 30423 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.198.2
I20260925 22:48:43.642606 30423 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.198.20:41677
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790376259650642-11032-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790376259650642-11032-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790376259650642-11032-0/krb5kdc/kudu_127.10.198.2.keytab
--principal=kudu/127.10.198.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.198.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790376259650642-11032-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.198.2
--webserver_port=0
--tserver_master_addrs=127.10.198.62:38175
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30423
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.198.2
--log_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790376259650642-11032-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:09:16 UTC on ccycloud.quasar-afmhyw.root.comops.site
build id 4673379
I20260925 22:48:43.643961 30423 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:48:43.645072 30423 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:48:43.655577 30428 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:48:43.655652 30423 server_base.cc:1034] running on GCE node
W20260925 22:48:43.655582 30429 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:48:43.655601 30431 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:48:43.656436 30423 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:48:43.657151 30423 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:48:43.658336 30423 hybrid_clock.cc:648] HybridClock initialized: now 1790376523658319 us; error 34 us; skew 500 ppm
Sep 25 22:48:43 dist-test-slave-5vdr krb5kdc[30182](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376523, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.198.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:48:43.661443 30423 init.cc:377] Logged in from keytab as kudu/127.10.198.2@KRBTEST.COM (short username kudu)
I20260925 22:48:43.662480 30423 webserver.cc:460] Webserver started at http://127.10.198.2:42775/ using document root <none> and password file <none>
I20260925 22:48:43.663130 30423 fs_manager.cc:359] Metadata directory not provided
I20260925 22:48:43.663237 30423 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:48:43.663479 30423 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:48:43.665843 30423 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790376259650642-11032-0/minicluster-data/ts-1/data/instance:
uuid: "7a0f7a509e4e43f8a8b3b102a5a3cbc6"
format_stamp: "Formatted at 2026-09-25 22:48:43 on dist-test-slave-5vdr"
server_key: "17a79d869183a16a937929b7ae89a410"
server_key_iv: "3401ee5eeea6da0e03b033eb488cb66e"
server_key_version: "encryptionkey@0"
I20260925 22:48:43.666477 30423 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790376259650642-11032-0/minicluster-data/ts-1/wal/instance:
uuid: "7a0f7a509e4e43f8a8b3b102a5a3cbc6"
format_stamp: "Formatted at 2026-09-25 22:48:43 on dist-test-slave-5vdr"
server_key: "17a79d869183a16a937929b7ae89a410"
server_key_iv: "3401ee5eeea6da0e03b033eb488cb66e"
server_key_version: "encryptionkey@0"
I20260925 22:48:43.670701 30423 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260925 22:48:43.673524 30438 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:48:43.674896 30423 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260925 22:48:43.675057 30423 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790376259650642-11032-0/minicluster-data/ts-1/data,/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790376259650642-11032-0/minicluster-data/ts-1/wal
uuid: "7a0f7a509e4e43f8a8b3b102a5a3cbc6"
format_stamp: "Formatted at 2026-09-25 22:48:43 on dist-test-slave-5vdr"
server_key: "17a79d869183a16a937929b7ae89a410"
server_key_iv: "3401ee5eeea6da0e03b033eb488cb66e"
server_key_version: "encryptionkey@0"
I20260925 22:48:43.675200 30423 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790376259650642-11032-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790376259650642-11032-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790376259650642-11032-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:48:43.695461 30423 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:48:43.699075 30423 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:48:43.699302 30423 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:48:43.700055 30423 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 22:48:43.700989 30423 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:48:43.701066 30423 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:48:43.701141 30423 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:48:43.701187 30423 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 22:48:43.711078 30423 rpc_server.cc:307] RPC server started. Bound to: 127.10.198.2:34447
I20260925 22:48:43.711095 30551 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.198.2:34447 every 8 connection(s)
I20260925 22:48:43.712185 30423 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790376259650642-11032-0/minicluster-data/ts-1/data/info.pb
I20260925 22:48:43.714763 11032 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrWXm8X/build/debug/bin/kudu as pid 30423
I20260925 22:48:43.714870 11032 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790376259650642-11032-0/minicluster-data/ts-1/wal/instance
I20260925 22:48:43.715222 11032 external_mini_cluster.cc:1428] Setting key 3d8db7acbba98b40b953039d84a38e3a
Sep 25 22:48:43 dist-test-slave-5vdr krb5kdc[30182](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376523, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.198.2@KRBTEST.COM for kudu/127.10.198.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.10.198.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.198.3@KRBTEST.COM" created.
I20260925 22:48:43.729460 30266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:48:43.714143 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.198.2:57349 (local address 127.10.198.62:38175)
0925 22:48:43.714297 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0925 22:48:43.714301 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:48:43.715463 (+ 1162us) server_negotiation.cc:407] Connection header received
0925 22:48:43.716962 (+ 1499us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:48:43.716965 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:48:43.717011 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:48:43.717075 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:48:43.718901 (+ 1826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:48:43.719791 (+ 890us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:48:43.721797 (+ 2006us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:48:43.722040 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:48:43.722137 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:48:43.725491 (+ 3354us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:48:43.725495 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:48:43.725498 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:48:43.725539 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:48:43.727134 (+ 1595us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:48:43.727730 (+ 596us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:48:43.727733 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:48:43.727734 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:48:43.727818 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:48:43.728249 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:48:43.728252 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:48:43.728254 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:48:43.728461 (+ 207us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:48:43.728547 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0925 22:48:43.729180 (+ 633us) server_negotiation.cc:299] Negotiation successful
0925 22:48:43.729324 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260925 22:48:43.730758 30554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:48:43.714605 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.198.62:38175 (local address 127.10.198.2:57349)
0925 22:48:43.715256 (+ 651us) negotiation.cc:107] Waiting for socket to connect
0925 22:48:43.715327 (+ 71us) client_negotiation.cc:174] Beginning negotiation
0925 22:48:43.716691 (+ 1364us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:48:43.717306 (+ 615us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:48:43.717317 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:48:43.717904 (+ 587us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:48:43.718689 (+ 785us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:48:43.718717 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:48:43.719959 (+ 1242us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:48:43.719961 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:48:43.721614 (+ 1653us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:48:43.721633 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:48:43.722242 (+ 609us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:48:43.722247 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:48:43.722514 (+ 267us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:48:43.723255 (+ 741us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:48:43.723284 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:48:43.725177 (+ 1893us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:48:43.727276 (+ 2099us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:48:43.727282 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:48:43.727300 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:48:43.727622 (+ 322us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:48:43.727979 (+ 357us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:48:43.727987 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:48:43.727988 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:48:43.728163 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:48:43.728565 (+ 402us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:48:43.728571 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:48:43.728850 (+ 279us) client_negotiation.cc:769] Sending connection context
0925 22:48:43.729094 (+ 244us) client_negotiation.cc:240] Negotiation successful
0925 22:48:43.729518 (+ 424us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":399,"thread_start_us":166,"threads_started":1}
I20260925 22:48:43.732242 30552 heartbeater.cc:347] Connected to a master server at 127.10.198.62:38175
I20260925 22:48:43.732573 30552 heartbeater.cc:464] Registering TS with master...
I20260925 22:48:43.733264 30552 heartbeater.cc:511] Master 127.10.198.62:38175 requested a full tablet report, sending...
I20260925 22:48:43.734719 30227 ts_manager.cc:194] Registered new tserver with Master: 7a0f7a509e4e43f8a8b3b102a5a3cbc6 (127.10.198.2:34447)
I20260925 22:48:43.735378 30227 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.198.2@KRBTEST.COM'} at 127.10.198.2:57349
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.198.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790376259650642-11032-0/krb5kdc/kudu_127.10.198.3.keytab.
Entry for principal kudu/127.10.198.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790376259650642-11032-0/krb5kdc/kudu_127.10.198.3.keytab.
I20260925 22:48:43.746433 11032 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrWXm8X/build/debug/bin/kudu
/tmp/dist-test-taskrWXm8X/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790376259650642-11032-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790376259650642-11032-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790376259650642-11032-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790376259650642-11032-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.198.3:0
--local_ip_for_outbound_sockets=127.10.198.3
--webserver_interface=127.10.198.3
--webserver_port=0
--tserver_master_addrs=127.10.198.62:38175
--keytab_file=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790376259650642-11032-0/krb5kdc/kudu_127.10.198.3.keytab
--principal=kudu/127.10.198.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.198.20:41677
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790376259650642-11032-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790376259650642-11032-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790376259650642-11032-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:48:43.866285 30557 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:48:43.866632 30557 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:48:43.866716 30557 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:48:43.871140 30557 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:48:43.871282 30557 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:48:43.871397 30557 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.198.3
I20260925 22:48:43.877251 30557 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.198.20:41677
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790376259650642-11032-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790376259650642-11032-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790376259650642-11032-0/krb5kdc/kudu_127.10.198.3.keytab
--principal=kudu/127.10.198.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.198.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790376259650642-11032-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.198.3
--webserver_port=0
--tserver_master_addrs=127.10.198.62:38175
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30557
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.198.3
--log_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790376259650642-11032-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:09:16 UTC on ccycloud.quasar-afmhyw.root.comops.site
build id 4673379
I20260925 22:48:43.878533 30557 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:48:43.879658 30557 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:48:43.890297 30562 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:48:43.890300 30563 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:48:43.890398 30557 server_base.cc:1034] running on GCE node
W20260925 22:48:43.890300 30565 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:48:43.891080 30557 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:48:43.891885 30557 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:48:43.893085 30557 hybrid_clock.cc:648] HybridClock initialized: now 1790376523893066 us; error 34 us; skew 500 ppm
Sep 25 22:48:43 dist-test-slave-5vdr krb5kdc[30182](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376523, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.198.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:48:43.896567 30557 init.cc:377] Logged in from keytab as kudu/127.10.198.3@KRBTEST.COM (short username kudu)
I20260925 22:48:43.897629 30557 webserver.cc:460] Webserver started at http://127.10.198.3:38865/ using document root <none> and password file <none>
I20260925 22:48:43.898276 30557 fs_manager.cc:359] Metadata directory not provided
I20260925 22:48:43.898382 30557 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:48:43.898636 30557 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:48:43.900992 30557 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790376259650642-11032-0/minicluster-data/ts-2/data/instance:
uuid: "a47a444c1ac2473bb9f704f8a69be7f5"
format_stamp: "Formatted at 2026-09-25 22:48:43 on dist-test-slave-5vdr"
server_key: "e12080895dd8ada6d2139ac46546089a"
server_key_iv: "b78c44a16802a9c7e132a1eadcfeb43a"
server_key_version: "encryptionkey@0"
I20260925 22:48:43.901614 30557 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790376259650642-11032-0/minicluster-data/ts-2/wal/instance:
uuid: "a47a444c1ac2473bb9f704f8a69be7f5"
format_stamp: "Formatted at 2026-09-25 22:48:43 on dist-test-slave-5vdr"
server_key: "e12080895dd8ada6d2139ac46546089a"
server_key_iv: "b78c44a16802a9c7e132a1eadcfeb43a"
server_key_version: "encryptionkey@0"
I20260925 22:48:43.906101 30557 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260925 22:48:43.908959 30572 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:48:43.910372 30557 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260925 22:48:43.910547 30557 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790376259650642-11032-0/minicluster-data/ts-2/data,/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790376259650642-11032-0/minicluster-data/ts-2/wal
uuid: "a47a444c1ac2473bb9f704f8a69be7f5"
format_stamp: "Formatted at 2026-09-25 22:48:43 on dist-test-slave-5vdr"
server_key: "e12080895dd8ada6d2139ac46546089a"
server_key_iv: "b78c44a16802a9c7e132a1eadcfeb43a"
server_key_version: "encryptionkey@0"
I20260925 22:48:43.910691 30557 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790376259650642-11032-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790376259650642-11032-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790376259650642-11032-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:48:43.922660 30557 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:48:43.926458 30557 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:48:43.926695 30557 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:48:43.927357 30557 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 22:48:43.928345 30557 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:48:43.928396 30557 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:48:43.928440 30557 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:48:43.928457 30557 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 22:48:43.938241 30557 rpc_server.cc:307] RPC server started. Bound to: 127.10.198.3:40601
I20260925 22:48:43.938262 30685 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.198.3:40601 every 8 connection(s)
I20260925 22:48:43.939359 30557 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790376259650642-11032-0/minicluster-data/ts-2/data/info.pb
I20260925 22:48:43.944285 11032 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrWXm8X/build/debug/bin/kudu as pid 30557
I20260925 22:48:43.944447 11032 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790376259650642-11032-0/minicluster-data/ts-2/wal/instance
I20260925 22:48:43.944743 11032 external_mini_cluster.cc:1428] Setting key cb0aaaa377f2878cf839b0ee4f6c22b0
Sep 25 22:48:43 dist-test-slave-5vdr krb5kdc[30182](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376523, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.198.3@KRBTEST.COM for kudu/127.10.198.62@KRBTEST.COM
I20260925 22:48:43.954442 30266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:48:43.941422 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.198.3:50735 (local address 127.10.198.62:38175)
0925 22:48:43.941575 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0925 22:48:43.941579 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:48:43.942849 (+ 1270us) server_negotiation.cc:407] Connection header received
0925 22:48:43.943926 (+ 1077us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:48:43.943930 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:48:43.943992 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:48:43.944086 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:48:43.945658 (+ 1572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:48:43.946381 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:48:43.947744 (+ 1363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:48:43.948072 (+ 328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:48:43.948190 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:48:43.950844 (+ 2654us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:48:43.950847 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:48:43.950849 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:48:43.950876 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:48:43.952208 (+ 1332us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:48:43.952795 (+ 587us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:48:43.952803 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:48:43.952805 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:48:43.952886 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:48:43.953282 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:48:43.953285 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:48:43.953287 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:48:43.953488 (+ 201us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:48:43.953594 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0925 22:48:43.954141 (+ 547us) server_negotiation.cc:299] Negotiation successful
0925 22:48:43.954272 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260925 22:48:43.955338 30688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:48:43.941901 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.198.62:38175 (local address 127.10.198.3:50735)
0925 22:48:43.942669 (+ 768us) negotiation.cc:107] Waiting for socket to connect
0925 22:48:43.942720 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0925 22:48:43.943630 (+ 910us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:48:43.944299 (+ 669us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:48:43.944306 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:48:43.944809 (+ 503us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:48:43.945499 (+ 690us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:48:43.945516 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:48:43.946537 (+ 1021us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:48:43.946541 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:48:43.947591 (+ 1050us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:48:43.947603 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:48:43.948249 (+ 646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:48:43.948253 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:48:43.948442 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:48:43.949067 (+ 625us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:48:43.949085 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:48:43.950627 (+ 1542us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:48:43.952342 (+ 1715us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:48:43.952349 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:48:43.952369 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:48:43.952658 (+ 289us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:48:43.952993 (+ 335us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:48:43.952996 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:48:43.952998 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:48:43.953174 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:48:43.953607 (+ 433us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:48:43.953617 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:48:43.953892 (+ 275us) client_negotiation.cc:769] Sending connection context
0925 22:48:43.954100 (+ 208us) client_negotiation.cc:240] Negotiation successful
0925 22:48:43.954461 (+ 361us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":553,"thread_start_us":152,"threads_started":1}
I20260925 22:48:43.956643 30686 heartbeater.cc:347] Connected to a master server at 127.10.198.62:38175
I20260925 22:48:43.956910 30686 heartbeater.cc:464] Registering TS with master...
I20260925 22:48:43.957558 30686 heartbeater.cc:511] Master 127.10.198.62:38175 requested a full tablet report, sending...
I20260925 22:48:43.958794 30227 ts_manager.cc:194] Registered new tserver with Master: a47a444c1ac2473bb9f704f8a69be7f5 (127.10.198.3:40601)
I20260925 22:48:43.959384 30227 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.198.3@KRBTEST.COM'} at 127.10.198.3:50735
I20260925 22:48:43.971159 11032 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 22:48:43.981261 30266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:48:43.973354 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55324 (local address 127.10.198.62:38175)
0925 22:48:43.973515 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0925 22:48:43.973520 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:48:43.973671 (+ 151us) server_negotiation.cc:407] Connection header received
0925 22:48:43.973791 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:48:43.973793 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:48:43.973839 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:48:43.973909 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:48:43.974619 (+ 710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:48:43.975168 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:48:43.976182 (+ 1014us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:48:43.976438 (+ 256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:48:43.976537 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:48:43.978090 (+ 1553us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:48:43.978093 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:48:43.978095 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:48:43.978124 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:48:43.979551 (+ 1427us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:48:43.980114 (+ 563us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:48:43.980117 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:48:43.980118 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:48:43.980201 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:48:43.980440 (+ 239us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:48:43.980442 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:48:43.980443 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:48:43.980681 (+ 238us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:48:43.980784 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0925 22:48:43.980967 (+ 183us) server_negotiation.cc:299] Negotiation successful
0925 22:48:43.981072 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260925 22:48:43.984679 30227 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55324:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 22:48:43.987931 30227 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 22:48:44.004144 30702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:48:43.998364 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53110 (local address 127.10.198.1:40675)
0925 22:48:43.998713 (+ 349us) server_negotiation.cc:206] Beginning negotiation
0925 22:48:43.998720 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 22:48:43.998822 (+ 102us) server_negotiation.cc:407] Connection header received
0925 22:48:43.998964 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:48:43.998969 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:48:43.999153 (+ 184us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:48:43.999314 (+ 161us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:48:43.999568 (+ 254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:48:44.001014 (+ 1446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:48:44.002211 (+ 1197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:48:44.003144 (+ 933us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:48:44.003260 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:48:44.003641 (+ 381us) server_negotiation.cc:1035] Waiting for connection context
0925 22:48:44.003877 (+ 236us) server_negotiation.cc:299] Negotiation successful
0925 22:48:44.003991 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":252,"thread_start_us":103,"threads_started":1}
I20260925 22:48:44.004132 30697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:48:43.997238 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.198.3:40601 (local address 127.0.0.1:37032)
0925 22:48:43.998058 (+ 820us) negotiation.cc:107] Waiting for socket to connect
0925 22:48:43.998119 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0925 22:48:43.998261 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:48:43.999111 (+ 850us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:48:43.999114 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:48:43.999130 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:48:43.999347 (+ 217us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:48:43.999355 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:48:44.000517 (+ 1162us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:48:44.000523 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:48:44.002303 (+ 1780us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:48:44.002314 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:48:44.003282 (+ 968us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:48:44.003285 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:48:44.003472 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:48:44.003562 (+ 90us) client_negotiation.cc:769] Sending connection context
0925 22:48:44.003690 (+ 128us) client_negotiation.cc:240] Negotiation successful
0925 22:48:44.003954 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":646,"mutex_wait_us":234,"thread_start_us":70,"threads_started":1}
I20260925 22:48:44.004412 30700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:48:43.997482 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41072 (local address 127.10.198.2:34447)
0925 22:48:43.998014 (+ 532us) server_negotiation.cc:206] Beginning negotiation
0925 22:48:43.998026 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 22:48:43.998404 (+ 378us) server_negotiation.cc:407] Connection header received
0925 22:48:43.998493 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:48:43.998498 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:48:43.998727 (+ 229us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:48:43.998884 (+ 157us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:48:43.999361 (+ 477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:48:44.000583 (+ 1222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:48:44.002453 (+ 1870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:48:44.003318 (+ 865us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:48:44.003893 (+ 575us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:48:44.004069 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0925 22:48:44.004123 (+ 54us) server_negotiation.cc:299] Negotiation successful
0925 22:48:44.004244 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":433,"thread_start_us":141,"threads_started":1}
I20260925 22:48:44.004842 30699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:48:43.997242 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.198.2:34447 (local address 127.0.0.1:41072)
0925 22:48:43.998144 (+ 902us) negotiation.cc:107] Waiting for socket to connect
0925 22:48:43.998158 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 22:48:43.998261 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:48:43.998985 (+ 724us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:48:43.998991 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:48:43.999010 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:48:43.999220 (+ 210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:48:43.999231 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:48:44.000727 (+ 1496us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:48:44.000730 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:48:44.002302 (+ 1572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:48:44.002314 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:48:44.003476 (+ 1162us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:48:44.003479 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:48:44.003714 (+ 235us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:48:44.003759 (+ 45us) client_negotiation.cc:769] Sending connection context
0925 22:48:44.004597 (+ 838us) client_negotiation.cc:240] Negotiation successful
0925 22:48:44.004703 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":845,"mutex_wait_us":341,"thread_start_us":106,"threads_started":1}
I20260925 22:48:44.004534 30701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:48:43.997932 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.198.1:40675 (local address 127.0.0.1:53110)
0925 22:48:43.998743 (+ 811us) negotiation.cc:107] Waiting for socket to connect
0925 22:48:43.998756 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 22:48:43.998855 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:48:43.999296 (+ 441us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:48:43.999302 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:48:43.999318 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:48:43.999448 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:48:43.999455 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:48:44.001151 (+ 1696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:48:44.001154 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:48:44.002058 (+ 904us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:48:44.002069 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:48:44.003474 (+ 1405us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:48:44.003478 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:48:44.003621 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:48:44.003651 (+ 30us) client_negotiation.cc:769] Sending connection context
0925 22:48:44.003732 (+ 81us) client_negotiation.cc:240] Negotiation successful
0925 22:48:44.004409 (+ 677us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":747,"mutex_wait_us":623,"thread_start_us":105,"threads_started":1}
I20260925 22:48:44.005033 30698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:48:43.997342 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37032 (local address 127.10.198.3:40601)
0925 22:48:43.998564 (+ 1222us) server_negotiation.cc:206] Beginning negotiation
0925 22:48:43.998575 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 22:48:43.998595 (+ 20us) server_negotiation.cc:407] Connection header received
0925 22:48:43.998656 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:48:43.998660 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:48:43.998830 (+ 170us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:48:43.998951 (+ 121us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:48:43.999491 (+ 540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:48:44.000378 (+ 887us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:48:44.002475 (+ 2097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:48:44.003132 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:48:44.003263 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:48:44.003436 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0925 22:48:44.003792 (+ 356us) server_negotiation.cc:299] Negotiation successful
0925 22:48:44.004859 (+ 1067us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1122,"thread_start_us":106,"threads_started":1}
I20260925 22:48:44.007807 30352 tablet_service.cc:1467] Processing CreateTablet for tablet 6a9268892cb845109855ac6263b59378 (DEFAULT_TABLE table=test-table [id=5286bd01546c40879a0be2b1e7990f6a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 22:48:44.007807 30486 tablet_service.cc:1467] Processing CreateTablet for tablet 6a9268892cb845109855ac6263b59378 (DEFAULT_TABLE table=test-table [id=5286bd01546c40879a0be2b1e7990f6a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 22:48:44.009228 30486 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6a9268892cb845109855ac6263b59378. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:48:44.009238 30352 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6a9268892cb845109855ac6263b59378. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:48:44.010414 30620 tablet_service.cc:1467] Processing CreateTablet for tablet 6a9268892cb845109855ac6263b59378 (DEFAULT_TABLE table=test-table [id=5286bd01546c40879a0be2b1e7990f6a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 22:48:44.011673 30620 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6a9268892cb845109855ac6263b59378. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:48:44.016619 30703 tablet_bootstrap.cc:492] T 6a9268892cb845109855ac6263b59378 P cd5b689f63d04cbebf9b88202711232d: Bootstrap starting.
I20260925 22:48:44.018352 30704 tablet_bootstrap.cc:492] T 6a9268892cb845109855ac6263b59378 P 7a0f7a509e4e43f8a8b3b102a5a3cbc6: Bootstrap starting.
I20260925 22:48:44.019244 30703 tablet_bootstrap.cc:654] T 6a9268892cb845109855ac6263b59378 P cd5b689f63d04cbebf9b88202711232d: Neither blocks nor log segments found. Creating new log.
I20260925 22:48:44.020304 30703 log.cc:824] T 6a9268892cb845109855ac6263b59378 P cd5b689f63d04cbebf9b88202711232d: Log is configured to *not* fsync() on all Append() calls
I20260925 22:48:44.021344 30704 tablet_bootstrap.cc:654] T 6a9268892cb845109855ac6263b59378 P 7a0f7a509e4e43f8a8b3b102a5a3cbc6: Neither blocks nor log segments found. Creating new log.
I20260925 22:48:44.021324 30705 tablet_bootstrap.cc:492] T 6a9268892cb845109855ac6263b59378 P a47a444c1ac2473bb9f704f8a69be7f5: Bootstrap starting.
I20260925 22:48:44.022351 30704 log.cc:824] T 6a9268892cb845109855ac6263b59378 P 7a0f7a509e4e43f8a8b3b102a5a3cbc6: Log is configured to *not* fsync() on all Append() calls
I20260925 22:48:44.022944 30703 tablet_bootstrap.cc:492] T 6a9268892cb845109855ac6263b59378 P cd5b689f63d04cbebf9b88202711232d: No bootstrap required, opened a new log
I20260925 22:48:44.023185 30703 ts_tablet_manager.cc:1397] T 6a9268892cb845109855ac6263b59378 P cd5b689f63d04cbebf9b88202711232d: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260925 22:48:44.023940 30705 tablet_bootstrap.cc:654] T 6a9268892cb845109855ac6263b59378 P a47a444c1ac2473bb9f704f8a69be7f5: Neither blocks nor log segments found. Creating new log.
I20260925 22:48:44.024904 30704 tablet_bootstrap.cc:492] T 6a9268892cb845109855ac6263b59378 P 7a0f7a509e4e43f8a8b3b102a5a3cbc6: No bootstrap required, opened a new log
I20260925 22:48:44.025126 30704 ts_tablet_manager.cc:1397] T 6a9268892cb845109855ac6263b59378 P 7a0f7a509e4e43f8a8b3b102a5a3cbc6: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260925 22:48:44.025319 30705 log.cc:824] T 6a9268892cb845109855ac6263b59378 P a47a444c1ac2473bb9f704f8a69be7f5: Log is configured to *not* fsync() on all Append() calls
I20260925 22:48:44.027812 30705 tablet_bootstrap.cc:492] T 6a9268892cb845109855ac6263b59378 P a47a444c1ac2473bb9f704f8a69be7f5: No bootstrap required, opened a new log
I20260925 22:48:44.028087 30705 ts_tablet_manager.cc:1397] T 6a9268892cb845109855ac6263b59378 P a47a444c1ac2473bb9f704f8a69be7f5: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.006s
I20260925 22:48:44.027881 30703 raft_consensus.cc:348] T 6a9268892cb845109855ac6263b59378 P cd5b689f63d04cbebf9b88202711232d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a47a444c1ac2473bb9f704f8a69be7f5" member_type: VOTER last_known_addr { host: "127.10.198.3" port: 40601 } } peers { permanent_uuid: "7a0f7a509e4e43f8a8b3b102a5a3cbc6" member_type: VOTER last_known_addr { host: "127.10.198.2" port: 34447 } } peers { permanent_uuid: "cd5b689f63d04cbebf9b88202711232d" member_type: VOTER last_known_addr { host: "127.10.198.1" port: 40675 } }
I20260925 22:48:44.028224 30703 raft_consensus.cc:374] T 6a9268892cb845109855ac6263b59378 P cd5b689f63d04cbebf9b88202711232d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:48:44.028285 30703 raft_consensus.cc:729] T 6a9268892cb845109855ac6263b59378 P cd5b689f63d04cbebf9b88202711232d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd5b689f63d04cbebf9b88202711232d, State: Initialized, Role: FOLLOWER
I20260925 22:48:44.028800 30703 consensus_queue.cc:260] T 6a9268892cb845109855ac6263b59378 P cd5b689f63d04cbebf9b88202711232d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a47a444c1ac2473bb9f704f8a69be7f5" member_type: VOTER last_known_addr { host: "127.10.198.3" port: 40601 } } peers { permanent_uuid: "7a0f7a509e4e43f8a8b3b102a5a3cbc6" member_type: VOTER last_known_addr { host: "127.10.198.2" port: 34447 } } peers { permanent_uuid: "cd5b689f63d04cbebf9b88202711232d" member_type: VOTER last_known_addr { host: "127.10.198.1" port: 40675 } }
I20260925 22:48:44.029294 30704 raft_consensus.cc:348] T 6a9268892cb845109855ac6263b59378 P 7a0f7a509e4e43f8a8b3b102a5a3cbc6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a47a444c1ac2473bb9f704f8a69be7f5" member_type: VOTER last_known_addr { host: "127.10.198.3" port: 40601 } } peers { permanent_uuid: "7a0f7a509e4e43f8a8b3b102a5a3cbc6" member_type: VOTER last_known_addr { host: "127.10.198.2" port: 34447 } } peers { permanent_uuid: "cd5b689f63d04cbebf9b88202711232d" member_type: VOTER last_known_addr { host: "127.10.198.1" port: 40675 } }
I20260925 22:48:44.029573 30418 heartbeater.cc:503] Master 127.10.198.62:38175 was elected leader, sending a full tablet report...
I20260925 22:48:44.029611 30704 raft_consensus.cc:374] T 6a9268892cb845109855ac6263b59378 P 7a0f7a509e4e43f8a8b3b102a5a3cbc6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:48:44.029666 30704 raft_consensus.cc:729] T 6a9268892cb845109855ac6263b59378 P 7a0f7a509e4e43f8a8b3b102a5a3cbc6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a0f7a509e4e43f8a8b3b102a5a3cbc6, State: Initialized, Role: FOLLOWER
I20260925 22:48:44.030169 30704 consensus_queue.cc:260] T 6a9268892cb845109855ac6263b59378 P 7a0f7a509e4e43f8a8b3b102a5a3cbc6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a47a444c1ac2473bb9f704f8a69be7f5" member_type: VOTER last_known_addr { host: "127.10.198.3" port: 40601 } } peers { permanent_uuid: "7a0f7a509e4e43f8a8b3b102a5a3cbc6" member_type: VOTER last_known_addr { host: "127.10.198.2" port: 34447 } } peers { permanent_uuid: "cd5b689f63d04cbebf9b88202711232d" member_type: VOTER last_known_addr { host: "127.10.198.1" port: 40675 } }
I20260925 22:48:44.030938 30552 heartbeater.cc:503] Master 127.10.198.62:38175 was elected leader, sending a full tablet report...
I20260925 22:48:44.031363 30704 ts_tablet_manager.cc:1428] T 6a9268892cb845109855ac6263b59378 P 7a0f7a509e4e43f8a8b3b102a5a3cbc6: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260925 22:48:44.031559 30703 ts_tablet_manager.cc:1428] T 6a9268892cb845109855ac6263b59378 P cd5b689f63d04cbebf9b88202711232d: Time spent starting tablet: real 0.008s user 0.008s sys 0.000s
I20260925 22:48:44.033787 30705 raft_consensus.cc:348] T 6a9268892cb845109855ac6263b59378 P a47a444c1ac2473bb9f704f8a69be7f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a47a444c1ac2473bb9f704f8a69be7f5" member_type: VOTER last_known_addr { host: "127.10.198.3" port: 40601 } } peers { permanent_uuid: "7a0f7a509e4e43f8a8b3b102a5a3cbc6" member_type: VOTER last_known_addr { host: "127.10.198.2" port: 34447 } } peers { permanent_uuid: "cd5b689f63d04cbebf9b88202711232d" member_type: VOTER last_known_addr { host: "127.10.198.1" port: 40675 } }
I20260925 22:48:44.034130 30705 raft_consensus.cc:374] T 6a9268892cb845109855ac6263b59378 P a47a444c1ac2473bb9f704f8a69be7f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:48:44.034193 30705 raft_consensus.cc:729] T 6a9268892cb845109855ac6263b59378 P a47a444c1ac2473bb9f704f8a69be7f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a47a444c1ac2473bb9f704f8a69be7f5, State: Initialized, Role: FOLLOWER
I20260925 22:48:44.034749 30705 consensus_queue.cc:260] T 6a9268892cb845109855ac6263b59378 P a47a444c1ac2473bb9f704f8a69be7f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a47a444c1ac2473bb9f704f8a69be7f5" member_type: VOTER last_known_addr { host: "127.10.198.3" port: 40601 } } peers { permanent_uuid: "7a0f7a509e4e43f8a8b3b102a5a3cbc6" member_type: VOTER last_known_addr { host: "127.10.198.2" port: 34447 } } peers { permanent_uuid: "cd5b689f63d04cbebf9b88202711232d" member_type: VOTER last_known_addr { host: "127.10.198.1" port: 40675 } }
I20260925 22:48:44.035635 30686 heartbeater.cc:503] Master 127.10.198.62:38175 was elected leader, sending a full tablet report...
I20260925 22:48:44.037298 30705 ts_tablet_manager.cc:1428] T 6a9268892cb845109855ac6263b59378 P a47a444c1ac2473bb9f704f8a69be7f5: Time spent starting tablet: real 0.009s user 0.000s sys 0.006s
W20260925 22:48:44.190564 30687 tablet.cc:2369] T 6a9268892cb845109855ac6263b59378 P a47a444c1ac2473bb9f704f8a69be7f5: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 22:48:44.213426 30553 tablet.cc:2369] T 6a9268892cb845109855ac6263b59378 P 7a0f7a509e4e43f8a8b3b102a5a3cbc6: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 22:48:44.217967 30419 tablet.cc:2369] T 6a9268892cb845109855ac6263b59378 P cd5b689f63d04cbebf9b88202711232d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 22:48:44.221074 30710 raft_consensus.cc:482] T 6a9268892cb845109855ac6263b59378 P 7a0f7a509e4e43f8a8b3b102a5a3cbc6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 22:48:44.221360 30710 raft_consensus.cc:504] T 6a9268892cb845109855ac6263b59378 P 7a0f7a509e4e43f8a8b3b102a5a3cbc6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a47a444c1ac2473bb9f704f8a69be7f5" member_type: VOTER last_known_addr { host: "127.10.198.3" port: 40601 } } peers { permanent_uuid: "7a0f7a509e4e43f8a8b3b102a5a3cbc6" member_type: VOTER last_known_addr { host: "127.10.198.2" port: 34447 } } peers { permanent_uuid: "cd5b689f63d04cbebf9b88202711232d" member_type: VOTER last_known_addr { host: "127.10.198.1" port: 40675 } }
I20260925 22:48:44.222609 30710 leader_election.cc:288] T 6a9268892cb845109855ac6263b59378 P 7a0f7a509e4e43f8a8b3b102a5a3cbc6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a47a444c1ac2473bb9f704f8a69be7f5 (127.10.198.3:40601), cd5b689f63d04cbebf9b88202711232d (127.10.198.1:40675)
I20260925 22:48:44.226974 30554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:48:44.222836 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.198.3:40601 (local address 127.10.198.2:58367)
0925 22:48:44.222979 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0925 22:48:44.222999 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0925 22:48:44.223106 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:48:44.223496 (+ 390us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:48:44.223499 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:48:44.223520 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:48:44.223667 (+ 147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:48:44.223688 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:48:44.224555 (+ 867us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:48:44.224562 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:48:44.225445 (+ 883us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:48:44.225457 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:48:44.226320 (+ 863us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:48:44.226327 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:48:44.226451 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:48:44.226492 (+ 41us) client_negotiation.cc:769] Sending connection context
0925 22:48:44.226576 (+ 84us) client_negotiation.cc:240] Negotiation successful
0925 22:48:44.226786 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":52}
I20260925 22:48:44.227078 30698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:48:44.223073 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.198.2:58367 (local address 127.10.198.3:40601)
0925 22:48:44.223225 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0925 22:48:44.223229 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:48:44.223246 (+ 17us) server_negotiation.cc:407] Connection header received
0925 22:48:44.223311 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:48:44.223316 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:48:44.223372 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:48:44.223457 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:48:44.223804 (+ 347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:48:44.224430 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:48:44.225626 (+ 1196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:48:44.226162 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:48:44.226280 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:48:44.226328 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0925 22:48:44.226758 (+ 430us) server_negotiation.cc:299] Negotiation successful
0925 22:48:44.226905 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260925 22:48:44.227151 30702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:48:44.223090 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.198.2:37787 (local address 127.10.198.1:40675)
0925 22:48:44.223282 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0925 22:48:44.223286 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:48:44.223365 (+ 79us) server_negotiation.cc:407] Connection header received
0925 22:48:44.223499 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:48:44.223503 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:48:44.223551 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:48:44.223659 (+ 108us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:48:44.223997 (+ 338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:48:44.224518 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:48:44.225853 (+ 1335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:48:44.226436 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:48:44.226586 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:48:44.226635 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0925 22:48:44.226922 (+ 287us) server_negotiation.cc:299] Negotiation successful
0925 22:48:44.227020 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":103}
I20260925 22:48:44.227216 30712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:48:44.222917 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.198.1:40675 (local address 127.10.198.2:37787)
0925 22:48:44.223263 (+ 346us) negotiation.cc:107] Waiting for socket to connect
0925 22:48:44.223278 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 22:48:44.223388 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:48:44.223697 (+ 309us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:48:44.223700 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:48:44.223715 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:48:44.223894 (+ 179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:48:44.223903 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:48:44.224653 (+ 750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:48:44.224656 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:48:44.225674 (+ 1018us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:48:44.225686 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:48:44.226647 (+ 961us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:48:44.226652 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:48:44.226784 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:48:44.226811 (+ 27us) client_negotiation.cc:769] Sending connection context
0925 22:48:44.226897 (+ 86us) client_negotiation.cc:240] Negotiation successful
0925 22:48:44.226971 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":264,"thread_start_us":94,"threads_started":1}
I20260925 22:48:44.227916 30640 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a9268892cb845109855ac6263b59378" candidate_uuid: "7a0f7a509e4e43f8a8b3b102a5a3cbc6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a47a444c1ac2473bb9f704f8a69be7f5" is_pre_election: true
I20260925 22:48:44.227916 30372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a9268892cb845109855ac6263b59378" candidate_uuid: "7a0f7a509e4e43f8a8b3b102a5a3cbc6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd5b689f63d04cbebf9b88202711232d" is_pre_election: true
I20260925 22:48:44.228281 30372 raft_consensus.cc:2454] T 6a9268892cb845109855ac6263b59378 P cd5b689f63d04cbebf9b88202711232d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a0f7a509e4e43f8a8b3b102a5a3cbc6 in term 0.
I20260925 22:48:44.228281 30640 raft_consensus.cc:2454] T 6a9268892cb845109855ac6263b59378 P a47a444c1ac2473bb9f704f8a69be7f5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a0f7a509e4e43f8a8b3b102a5a3cbc6 in term 0.
I20260925 22:48:44.228914 30441 leader_election.cc:302] T 6a9268892cb845109855ac6263b59378 P 7a0f7a509e4e43f8a8b3b102a5a3cbc6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 7a0f7a509e4e43f8a8b3b102a5a3cbc6, a47a444c1ac2473bb9f704f8a69be7f5, cd5b689f63d04cbebf9b88202711232d; no voters:
I20260925 22:48:44.229285 30710 raft_consensus.cc:2781] T 6a9268892cb845109855ac6263b59378 P 7a0f7a509e4e43f8a8b3b102a5a3cbc6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 22:48:44.229403 30710 raft_consensus.cc:482] T 6a9268892cb845109855ac6263b59378 P 7a0f7a509e4e43f8a8b3b102a5a3cbc6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 22:48:44.229491 30710 raft_consensus.cc:3037] T 6a9268892cb845109855ac6263b59378 P 7a0f7a509e4e43f8a8b3b102a5a3cbc6 [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:48:44.230679 30710 raft_consensus.cc:504] T 6a9268892cb845109855ac6263b59378 P 7a0f7a509e4e43f8a8b3b102a5a3cbc6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a47a444c1ac2473bb9f704f8a69be7f5" member_type: VOTER last_known_addr { host: "127.10.198.3" port: 40601 } } peers { permanent_uuid: "7a0f7a509e4e43f8a8b3b102a5a3cbc6" member_type: VOTER last_known_addr { host: "127.10.198.2" port: 34447 } } peers { permanent_uuid: "cd5b689f63d04cbebf9b88202711232d" member_type: VOTER last_known_addr { host: "127.10.198.1" port: 40675 } }
I20260925 22:48:44.231114 30710 leader_election.cc:288] T 6a9268892cb845109855ac6263b59378 P 7a0f7a509e4e43f8a8b3b102a5a3cbc6 [CANDIDATE]: Term 1 election: Requested vote from peers a47a444c1ac2473bb9f704f8a69be7f5 (127.10.198.3:40601), cd5b689f63d04cbebf9b88202711232d (127.10.198.1:40675)
I20260925 22:48:44.231606 30640 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a9268892cb845109855ac6263b59378" candidate_uuid: "7a0f7a509e4e43f8a8b3b102a5a3cbc6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a47a444c1ac2473bb9f704f8a69be7f5"
I20260925 22:48:44.231612 30372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a9268892cb845109855ac6263b59378" candidate_uuid: "7a0f7a509e4e43f8a8b3b102a5a3cbc6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd5b689f63d04cbebf9b88202711232d"
I20260925 22:48:44.231791 30372 raft_consensus.cc:3037] T 6a9268892cb845109855ac6263b59378 P cd5b689f63d04cbebf9b88202711232d [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:48:44.231774 30640 raft_consensus.cc:3037] T 6a9268892cb845109855ac6263b59378 P a47a444c1ac2473bb9f704f8a69be7f5 [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:48:44.233194 30372 raft_consensus.cc:2454] T 6a9268892cb845109855ac6263b59378 P cd5b689f63d04cbebf9b88202711232d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a0f7a509e4e43f8a8b3b102a5a3cbc6 in term 1.
I20260925 22:48:44.233255 30640 raft_consensus.cc:2454] T 6a9268892cb845109855ac6263b59378 P a47a444c1ac2473bb9f704f8a69be7f5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a0f7a509e4e43f8a8b3b102a5a3cbc6 in term 1.
I20260925 22:48:44.233559 30441 leader_election.cc:302] T 6a9268892cb845109855ac6263b59378 P 7a0f7a509e4e43f8a8b3b102a5a3cbc6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a0f7a509e4e43f8a8b3b102a5a3cbc6, cd5b689f63d04cbebf9b88202711232d; no voters:
I20260925 22:48:44.233827 30710 raft_consensus.cc:2781] T 6a9268892cb845109855ac6263b59378 P 7a0f7a509e4e43f8a8b3b102a5a3cbc6 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 22:48:44.234139 30710 raft_consensus.cc:686] T 6a9268892cb845109855ac6263b59378 P 7a0f7a509e4e43f8a8b3b102a5a3cbc6 [term 1 LEADER]: Becoming Leader. State: Replica: 7a0f7a509e4e43f8a8b3b102a5a3cbc6, State: Running, Role: LEADER
I20260925 22:48:44.234578 30710 consensus_queue.cc:237] T 6a9268892cb845109855ac6263b59378 P 7a0f7a509e4e43f8a8b3b102a5a3cbc6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a47a444c1ac2473bb9f704f8a69be7f5" member_type: VOTER last_known_addr { host: "127.10.198.3" port: 40601 } } peers { permanent_uuid: "7a0f7a509e4e43f8a8b3b102a5a3cbc6" member_type: VOTER last_known_addr { host: "127.10.198.2" port: 34447 } } peers { permanent_uuid: "cd5b689f63d04cbebf9b88202711232d" member_type: VOTER last_known_addr { host: "127.10.198.1" port: 40675 } }
I20260925 22:48:44.238274 30227 catalog_manager.cc:5495] T 6a9268892cb845109855ac6263b59378 P 7a0f7a509e4e43f8a8b3b102a5a3cbc6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7a0f7a509e4e43f8a8b3b102a5a3cbc6 (127.10.198.2). New cstate: current_term: 1 leader_uuid: "7a0f7a509e4e43f8a8b3b102a5a3cbc6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a47a444c1ac2473bb9f704f8a69be7f5" member_type: VOTER last_known_addr { host: "127.10.198.3" port: 40601 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7a0f7a509e4e43f8a8b3b102a5a3cbc6" member_type: VOTER last_known_addr { host: "127.10.198.2" port: 34447 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cd5b689f63d04cbebf9b88202711232d" member_type: VOTER last_known_addr { host: "127.10.198.1" port: 40675 } health_report { overall_health: UNKNOWN } } }
I20260925 22:48:44.299598 30700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:48:44.296047 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41076 (local address 127.10.198.2:34447)
0925 22:48:44.296183 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0925 22:48:44.296187 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:48:44.296202 (+ 15us) server_negotiation.cc:407] Connection header received
0925 22:48:44.296284 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:48:44.296286 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:48:44.296344 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:48:44.296449 (+ 105us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 22:48:44.296672 (+ 223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:48:44.297180 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:48:44.298065 (+ 885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:48:44.298314 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:48:44.298405 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:48:44.298686 (+ 281us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 22:48:44.299188 (+ 502us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 22:48:44.299290 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0925 22:48:44.299385 (+ 95us) server_negotiation.cc:299] Negotiation successful
0925 22:48:44.299446 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260925 22:48:44.306402 30372 raft_consensus.cc:1261] T 6a9268892cb845109855ac6263b59378 P cd5b689f63d04cbebf9b88202711232d [term 1 FOLLOWER]: Refusing update from remote peer 7a0f7a509e4e43f8a8b3b102a5a3cbc6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 22:48:44.307232 30640 raft_consensus.cc:1261] T 6a9268892cb845109855ac6263b59378 P a47a444c1ac2473bb9f704f8a69be7f5 [term 1 FOLLOWER]: Refusing update from remote peer 7a0f7a509e4e43f8a8b3b102a5a3cbc6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 22:48:44.307294 30710 consensus_queue.cc:1035] T 6a9268892cb845109855ac6263b59378 P 7a0f7a509e4e43f8a8b3b102a5a3cbc6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd5b689f63d04cbebf9b88202711232d" member_type: VOTER last_known_addr { host: "127.10.198.1" port: 40675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 22:48:44.307986 30710 consensus_queue.cc:1035] T 6a9268892cb845109855ac6263b59378 P 7a0f7a509e4e43f8a8b3b102a5a3cbc6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a47a444c1ac2473bb9f704f8a69be7f5" member_type: VOTER last_known_addr { host: "127.10.198.3" port: 40601 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 22:48:44.317546 30716 mvcc.cc:205] Tried to move back new op lower bound from 7333382243550220288 to 7333382243267866624. Current Snapshot: MvccSnapshot[applied={T|T < 7333382243550220288}]
I20260925 22:48:44.317615 30717 mvcc.cc:205] Tried to move back new op lower bound from 7333382243550220288 to 7333382243267866624. Current Snapshot: MvccSnapshot[applied={T|T < 7333382243550220288}]
I20260925 22:48:44.317749 30718 mvcc.cc:205] Tried to move back new op lower bound from 7333382243550220288 to 7333382243267866624. Current Snapshot: MvccSnapshot[applied={T|T < 7333382243550220288}]
I20260925 22:48:44.327400 30226 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55324:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 22:48:44.327631 30226 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55324:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 22:48:44.330564 30226 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c25248a58bee4ebcaaab275864d564da: Sending DeleteTablet for 3 replicas of tablet 6a9268892cb845109855ac6263b59378
I20260925 22:48:44.331537 30486 tablet_service.cc:1514] Processing DeleteTablet for tablet 6a9268892cb845109855ac6263b59378 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 22:48:44 UTC) from {username='kudu', principal='kudu/127.10.198.62@KRBTEST.COM'} at 127.0.0.1:41072
I20260925 22:48:44.331537 30620 tablet_service.cc:1514] Processing DeleteTablet for tablet 6a9268892cb845109855ac6263b59378 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 22:48:44 UTC) from {username='kudu', principal='kudu/127.10.198.62@KRBTEST.COM'} at 127.0.0.1:37032
I20260925 22:48:44.331600 30352 tablet_service.cc:1514] Processing DeleteTablet for tablet 6a9268892cb845109855ac6263b59378 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 22:48:44 UTC) from {username='kudu', principal='kudu/127.10.198.62@KRBTEST.COM'} at 127.0.0.1:53110
I20260925 22:48:44.332381 30728 tablet_replica.cc:330] T 6a9268892cb845109855ac6263b59378 P 7a0f7a509e4e43f8a8b3b102a5a3cbc6: stopping tablet replica
I20260925 22:48:44.332809 30728 raft_consensus.cc:2229] T 6a9268892cb845109855ac6263b59378 P 7a0f7a509e4e43f8a8b3b102a5a3cbc6 [term 1 LEADER]: Raft consensus shutting down.
I20260925 22:48:44.332989 11032 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrWXm8X/build/debug/bin/kudu with pid 30289
I20260925 22:48:44.334136 30728 raft_consensus.cc:2258] T 6a9268892cb845109855ac6263b59378 P 7a0f7a509e4e43f8a8b3b102a5a3cbc6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 22:48:44.334611 30726 tablet_replica.cc:330] T 6a9268892cb845109855ac6263b59378 P a47a444c1ac2473bb9f704f8a69be7f5: stopping tablet replica
I20260925 22:48:44.335002 30726 raft_consensus.cc:2229] T 6a9268892cb845109855ac6263b59378 P a47a444c1ac2473bb9f704f8a69be7f5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 22:48:44.335347 30726 raft_consensus.cc:2258] T 6a9268892cb845109855ac6263b59378 P a47a444c1ac2473bb9f704f8a69be7f5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 22:48:44.337683 30728 ts_tablet_manager.cc:1905] T 6a9268892cb845109855ac6263b59378 P 7a0f7a509e4e43f8a8b3b102a5a3cbc6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 22:48:44.341542 30726 ts_tablet_manager.cc:1905] T 6a9268892cb845109855ac6263b59378 P a47a444c1ac2473bb9f704f8a69be7f5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 22:48:44.341717 30728 ts_tablet_manager.cc:1918] T 6a9268892cb845109855ac6263b59378 P 7a0f7a509e4e43f8a8b3b102a5a3cbc6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
W20260925 22:48:44.342121 30213 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.10.198.1:40675 (error 108)
I20260925 22:48:44.342273 11032 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrWXm8X/build/debug/bin/kudu with pid 30423
I20260925 22:48:44.345463 30726 ts_tablet_manager.cc:1918] T 6a9268892cb845109855ac6263b59378 P a47a444c1ac2473bb9f704f8a69be7f5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 22:48:44.345758 30726 log.cc:1196] T 6a9268892cb845109855ac6263b59378 P a47a444c1ac2473bb9f704f8a69be7f5: Deleting WAL directory at /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790376259650642-11032-0/minicluster-data/ts-2/wal/wals/6a9268892cb845109855ac6263b59378
I20260925 22:48:44.345841 30701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:48:44.345468 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.198.1:40675 (local address 127.0.0.1:53126)
0925 22:48:44.345645 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0925 22:48:44.345735 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.198.1:40675: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
I20260925 22:48:44.346136 30726 ts_tablet_manager.cc:1939] T 6a9268892cb845109855ac6263b59378 P a47a444c1ac2473bb9f704f8a69be7f5: Deleting consensus metadata
W20260925 22:48:44.346087 30213 catalog_manager.cc:4584] TS cd5b689f63d04cbebf9b88202711232d (127.10.198.1:40675): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 6a9268892cb845109855ac6263b59378: Network error: Client connection negotiation failed: client connection to 127.10.198.1:40675: connect: Connection refused (error 111)
I20260925 22:48:44.347425 30211 catalog_manager.cc:4841] TS a47a444c1ac2473bb9f704f8a69be7f5 (127.10.198.3:40601): tablet 6a9268892cb845109855ac6263b59378 (table test-table [id=5286bd01546c40879a0be2b1e7990f6a]) successfully deleted
I20260925 22:48:44.349351 11032 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrWXm8X/build/debug/bin/kudu with pid 30557
I20260925 22:48:44.350229 30701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:48:44.349554 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.198.2:34447 (local address 127.0.0.1:41086)
0925 22:48:44.350056 (+ 502us) negotiation.cc:107] Waiting for socket to connect
0925 22:48:44.350126 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.198.2:34447: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":395}
I20260925 22:48:44.355671 11032 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrWXm8X/build/debug/bin/kudu with pid 30195
2026-09-25T22:48:44Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (2550 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-09-25T22:48:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T22:48:44Z Disabled control of system clock
I20260925 22:48:44.387619 11032 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrWXm8X/build/debug/bin/kudu
/tmp/dist-test-taskrWXm8X/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790376259650642-11032-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790376259650642-11032-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790376259650642-11032-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790376259650642-11032-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.198.62:39021
--webserver_interface=127.10.198.62
--webserver_port=0
--builtin_ntp_servers=127.10.198.20:36321
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.198.62:39021
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260925 22:48:44.506424 30733 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:48:44.506755 30733 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:48:44.506794 30733 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:48:44.511053 30733 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 22:48:44.511147 30733 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:48:44.511163 30733 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:48:44.511181 30733 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 22:48:44.511197 30733 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 22:48:44.516520 30733 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.198.20:36321
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790376259650642-11032-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790376259650642-11032-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.198.62:39021
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.198.62:39021
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790376259650642-11032-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.198.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30733
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790376259650642-11032-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:09:16 UTC on ccycloud.quasar-afmhyw.root.comops.site
build id 4673379
I20260925 22:48:44.517696 30733 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:48:44.519134 30733 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:48:44.529070 30741 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:48:44.529070 30738 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:48:44.529090 30739 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:48:44.529537 30733 server_base.cc:1034] running on GCE node
I20260925 22:48:44.530231 30733 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:48:44.531702 30733 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:48:44.532907 30733 hybrid_clock.cc:648] HybridClock initialized: now 1790376524532886 us; error 61 us; skew 500 ppm
I20260925 22:48:44.535270 30733 webserver.cc:460] Webserver started at http://127.10.198.62:34461/ using document root <none> and password file <none>
I20260925 22:48:44.536038 30733 fs_manager.cc:359] Metadata directory not provided
I20260925 22:48:44.536123 30733 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:48:44.536372 30733 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:48:44.538689 30733 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790376259650642-11032-0/minicluster-data/master-0/data/instance:
uuid: "f5fd62008eaf471c908be628985ba2e9"
format_stamp: "Formatted at 2026-09-25 22:48:44 on dist-test-slave-5vdr"
server_key: "06b99655b8199a14e5272fdbbd9c786f"
server_key_iv: "01690e6d4a33e1285935b37c47462958"
server_key_version: "encryptionkey@0"
I20260925 22:48:44.539350 30733 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790376259650642-11032-0/minicluster-data/master-0/wal/instance:
uuid: "f5fd62008eaf471c908be628985ba2e9"
format_stamp: "Formatted at 2026-09-25 22:48:44 on dist-test-slave-5vdr"
server_key: "06b99655b8199a14e5272fdbbd9c786f"
server_key_iv: "01690e6d4a33e1285935b37c47462958"
server_key_version: "encryptionkey@0"
I20260925 22:48:44.543673 30733 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260925 22:48:44.546696 30747 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:48:44.548331 30733 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 22:48:44.548508 30733 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790376259650642-11032-0/minicluster-data/master-0/data,/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790376259650642-11032-0/minicluster-data/master-0/wal
uuid: "f5fd62008eaf471c908be628985ba2e9"
format_stamp: "Formatted at 2026-09-25 22:48:44 on dist-test-slave-5vdr"
server_key: "06b99655b8199a14e5272fdbbd9c786f"
server_key_iv: "01690e6d4a33e1285935b37c47462958"
server_key_version: "encryptionkey@0"
I20260925 22:48:44.548662 30733 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790376259650642-11032-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790376259650642-11032-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790376259650642-11032-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:48:44.568622 30733 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:48:44.569485 30733 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:48:44.569701 30733 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:48:44.577350 30733 rpc_server.cc:307] RPC server started. Bound to: 127.10.198.62:39021
I20260925 22:48:44.577359 30799 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.198.62:39021 every 8 connection(s)
I20260925 22:48:44.578503 30733 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790376259650642-11032-0/minicluster-data/master-0/data/info.pb
I20260925 22:48:44.581988 30800 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:48:44.584851 11032 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrWXm8X/build/debug/bin/kudu as pid 30733
I20260925 22:48:44.584965 11032 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790376259650642-11032-0/minicluster-data/master-0/wal/instance
I20260925 22:48:44.585301 11032 external_mini_cluster.cc:1428] Setting key 2c93bc7f9233b03ecf0d05f197b65245
I20260925 22:48:44.589071 30800 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f5fd62008eaf471c908be628985ba2e9: Bootstrap starting.
I20260925 22:48:44.592087 30800 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f5fd62008eaf471c908be628985ba2e9: Neither blocks nor log segments found. Creating new log.
I20260925 22:48:44.593060 30800 log.cc:824] T 00000000000000000000000000000000 P f5fd62008eaf471c908be628985ba2e9: Log is configured to *not* fsync() on all Append() calls
I20260925 22:48:44.595884 30800 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f5fd62008eaf471c908be628985ba2e9: No bootstrap required, opened a new log
I20260925 22:48:44.597007 30803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:48:44.586897 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49308 (local address 127.10.198.62:39021)
0925 22:48:44.587722 (+ 825us) server_negotiation.cc:206] Beginning negotiation
0925 22:48:44.587744 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0925 22:48:44.587794 (+ 50us) server_negotiation.cc:407] Connection header received
0925 22:48:44.588708 (+ 914us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:48:44.588731 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:48:44.589276 (+ 545us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:48:44.589674 (+ 398us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:48:44.590147 (+ 473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:48:44.591685 (+ 1538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:48:44.592746 (+ 1061us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:48:44.593490 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:48:44.593647 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:48:44.594617 (+ 970us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:48:44.594624 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:48:44.594642 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 22:48:44.595204 (+ 562us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:48:44.595287 (+ 83us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 22:48:44.595316 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:48:44.595430 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0925 22:48:44.595644 (+ 214us) server_negotiation.cc:299] Negotiation successful
0925 22:48:44.595990 (+ 346us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":321,"thread_start_us":125,"threads_started":1}
I20260925 22:48:44.600512 30800 raft_consensus.cc:348] T 00000000000000000000000000000000 P f5fd62008eaf471c908be628985ba2e9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f5fd62008eaf471c908be628985ba2e9" member_type: VOTER last_known_addr { host: "127.10.198.62" port: 39021 } }
I20260925 22:48:44.600802 30800 raft_consensus.cc:374] T 00000000000000000000000000000000 P f5fd62008eaf471c908be628985ba2e9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:48:44.600871 30800 raft_consensus.cc:729] T 00000000000000000000000000000000 P f5fd62008eaf471c908be628985ba2e9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f5fd62008eaf471c908be628985ba2e9, State: Initialized, Role: FOLLOWER
I20260925 22:48:44.601385 30800 consensus_queue.cc:260] T 00000000000000000000000000000000 P f5fd62008eaf471c908be628985ba2e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f5fd62008eaf471c908be628985ba2e9" member_type: VOTER last_known_addr { host: "127.10.198.62" port: 39021 } }
I20260925 22:48:44.601528 30800 raft_consensus.cc:388] T 00000000000000000000000000000000 P f5fd62008eaf471c908be628985ba2e9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 22:48:44.601581 30800 raft_consensus.cc:482] T 00000000000000000000000000000000 P f5fd62008eaf471c908be628985ba2e9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 22:48:44.601661 30800 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f5fd62008eaf471c908be628985ba2e9 [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:48:44.602909 30800 raft_consensus.cc:504] T 00000000000000000000000000000000 P f5fd62008eaf471c908be628985ba2e9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f5fd62008eaf471c908be628985ba2e9" member_type: VOTER last_known_addr { host: "127.10.198.62" port: 39021 } }
I20260925 22:48:44.603329 30800 leader_election.cc:302] T 00000000000000000000000000000000 P f5fd62008eaf471c908be628985ba2e9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f5fd62008eaf471c908be628985ba2e9; no voters:
I20260925 22:48:44.603660 30800 leader_election.cc:288] T 00000000000000000000000000000000 P f5fd62008eaf471c908be628985ba2e9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 22:48:44.603924 30805 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f5fd62008eaf471c908be628985ba2e9 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 22:48:44.604290 30805 raft_consensus.cc:686] T 00000000000000000000000000000000 P f5fd62008eaf471c908be628985ba2e9 [term 1 LEADER]: Becoming Leader. State: Replica: f5fd62008eaf471c908be628985ba2e9, State: Running, Role: LEADER
I20260925 22:48:44.604703 30805 consensus_queue.cc:237] T 00000000000000000000000000000000 P f5fd62008eaf471c908be628985ba2e9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f5fd62008eaf471c908be628985ba2e9" member_type: VOTER last_known_addr { host: "127.10.198.62" port: 39021 } }
I20260925 22:48:44.605192 30800 sys_catalog.cc:564] T 00000000000000000000000000000000 P f5fd62008eaf471c908be628985ba2e9 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 22:48:44.606473 30807 sys_catalog.cc:455] T 00000000000000000000000000000000 P f5fd62008eaf471c908be628985ba2e9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f5fd62008eaf471c908be628985ba2e9. Latest consensus state: current_term: 1 leader_uuid: "f5fd62008eaf471c908be628985ba2e9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f5fd62008eaf471c908be628985ba2e9" member_type: VOTER last_known_addr { host: "127.10.198.62" port: 39021 } } }
I20260925 22:48:44.606653 30807 sys_catalog.cc:458] T 00000000000000000000000000000000 P f5fd62008eaf471c908be628985ba2e9 [sys.catalog]: This master's current role is: LEADER
I20260925 22:48:44.607031 30806 sys_catalog.cc:455] T 00000000000000000000000000000000 P f5fd62008eaf471c908be628985ba2e9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f5fd62008eaf471c908be628985ba2e9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f5fd62008eaf471c908be628985ba2e9" member_type: VOTER last_known_addr { host: "127.10.198.62" port: 39021 } } }
I20260925 22:48:44.607125 30814 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 22:48:44.607141 30806 sys_catalog.cc:458] T 00000000000000000000000000000000 P f5fd62008eaf471c908be628985ba2e9 [sys.catalog]: This master's current role is: LEADER
I20260925 22:48:44.610733 30814 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 22:48:44.617245 30814 catalog_manager.cc:1347] Generated new cluster ID: 510a0a40beb848878d34abd86cc814a6
I20260925 22:48:44.617352 30814 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 22:48:44.629343 30814 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 22:48:44.630422 30814 catalog_manager.cc:1504] Loading token signing keys...
I20260925 22:48:44.640007 30814 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f5fd62008eaf471c908be628985ba2e9: Generated new TSK 0
I20260925 22:48:44.641064 30814 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 22:48:44.644291 11032 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrWXm8X/build/debug/bin/kudu
/tmp/dist-test-taskrWXm8X/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790376259650642-11032-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790376259650642-11032-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790376259650642-11032-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790376259650642-11032-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.198.1:0
--local_ip_for_outbound_sockets=127.10.198.1
--webserver_interface=127.10.198.1
--webserver_port=0
--tserver_master_addrs=127.10.198.62:39021
--builtin_ntp_servers=127.10.198.20:36321
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260925 22:48:44.767604 30824 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:48:44.768015 30824 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:48:44.768064 30824 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:48:44.772722 30824 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:48:44.772802 30824 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:48:44.772863 30824 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.198.1
I20260925 22:48:44.778196 30824 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.198.20:36321
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790376259650642-11032-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790376259650642-11032-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.198.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790376259650642-11032-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.198.1
--webserver_port=0
--tserver_master_addrs=127.10.198.62:39021
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30824
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.198.1
--log_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790376259650642-11032-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:09:16 UTC on ccycloud.quasar-afmhyw.root.comops.site
build id 4673379
I20260925 22:48:44.779367 30824 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:48:44.780958 30824 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:48:44.791531 30829 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:48:44.791543 30832 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:48:44.791745 30824 server_base.cc:1034] running on GCE node
W20260925 22:48:44.791541 30830 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:48:44.792435 30824 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:48:44.793205 30824 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:48:44.794451 30824 hybrid_clock.cc:648] HybridClock initialized: now 1790376524794432 us; error 36 us; skew 500 ppm
I20260925 22:48:44.796694 30824 webserver.cc:460] Webserver started at http://127.10.198.1:38295/ using document root <none> and password file <none>
I20260925 22:48:44.797384 30824 fs_manager.cc:359] Metadata directory not provided
I20260925 22:48:44.797499 30824 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:48:44.797758 30824 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:48:44.800158 30824 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790376259650642-11032-0/minicluster-data/ts-0/data/instance:
uuid: "40d68243fa5344f78b05bd14d8516a5e"
format_stamp: "Formatted at 2026-09-25 22:48:44 on dist-test-slave-5vdr"
server_key: "023ca29520eece3d855215085087d580"
server_key_iv: "ae419621ef8de51fa0942e4d894b8f05"
server_key_version: "encryptionkey@0"
I20260925 22:48:44.800896 30824 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790376259650642-11032-0/minicluster-data/ts-0/wal/instance:
uuid: "40d68243fa5344f78b05bd14d8516a5e"
format_stamp: "Formatted at 2026-09-25 22:48:44 on dist-test-slave-5vdr"
server_key: "023ca29520eece3d855215085087d580"
server_key_iv: "ae419621ef8de51fa0942e4d894b8f05"
server_key_version: "encryptionkey@0"
I20260925 22:48:44.805194 30824 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.004s
I20260925 22:48:44.808204 30838 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:48:44.809659 30824 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260925 22:48:44.809846 30824 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790376259650642-11032-0/minicluster-data/ts-0/data,/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790376259650642-11032-0/minicluster-data/ts-0/wal
uuid: "40d68243fa5344f78b05bd14d8516a5e"
format_stamp: "Formatted at 2026-09-25 22:48:44 on dist-test-slave-5vdr"
server_key: "023ca29520eece3d855215085087d580"
server_key_iv: "ae419621ef8de51fa0942e4d894b8f05"
server_key_version: "encryptionkey@0"
I20260925 22:48:44.809993 30824 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790376259650642-11032-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790376259650642-11032-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790376259650642-11032-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:48:44.862751 30824 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:48:44.863554 30824 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:48:44.863796 30824 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:48:44.864478 30824 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 22:48:44.865480 30824 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:48:44.865533 30824 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:48:44.865576 30824 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:48:44.865602 30824 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 22:48:44.875213 30824 rpc_server.cc:307] RPC server started. Bound to: 127.10.198.1:44215
I20260925 22:48:44.875233 30951 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.198.1:44215 every 8 connection(s)
I20260925 22:48:44.876317 30824 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790376259650642-11032-0/minicluster-data/ts-0/data/info.pb
I20260925 22:48:44.882349 11032 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrWXm8X/build/debug/bin/kudu as pid 30824
I20260925 22:48:44.882517 11032 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790376259650642-11032-0/minicluster-data/ts-0/wal/instance
I20260925 22:48:44.882817 11032 external_mini_cluster.cc:1428] Setting key 281688bf0ac4e417af783f227aadffaa
I20260925 22:48:44.884974 11032 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrWXm8X/build/debug/bin/kudu
/tmp/dist-test-taskrWXm8X/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790376259650642-11032-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790376259650642-11032-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790376259650642-11032-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790376259650642-11032-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.198.2:0
--local_ip_for_outbound_sockets=127.10.198.2
--webserver_interface=127.10.198.2
--webserver_port=0
--tserver_master_addrs=127.10.198.62:39021
--builtin_ntp_servers=127.10.198.20:36321
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260925 22:48:44.886687 30803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:48:44.878320 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.198.1:60453 (local address 127.10.198.62:39021)
0925 22:48:44.878480 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0925 22:48:44.878484 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:48:44.879634 (+ 1150us) server_negotiation.cc:407] Connection header received
0925 22:48:44.881191 (+ 1557us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:48:44.881195 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:48:44.881260 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:48:44.881378 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:48:44.882379 (+ 1001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:48:44.883086 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:48:44.884471 (+ 1385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:48:44.884711 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:48:44.884834 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:48:44.886145 (+ 1311us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:48:44.886149 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:48:44.886152 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 22:48:44.886181 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:48:44.886201 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 22:48:44.886209 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:48:44.886288 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0925 22:48:44.886466 (+ 178us) server_negotiation.cc:299] Negotiation successful
0925 22:48:44.886535 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260925 22:48:44.887673 30954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:48:44.878755 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.198.62:39021 (local address 127.10.198.1:60453)
0925 22:48:44.879465 (+ 710us) negotiation.cc:107] Waiting for socket to connect
0925 22:48:44.879512 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0925 22:48:44.880934 (+ 1422us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:48:44.881577 (+ 643us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:48:44.881585 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:48:44.881684 (+ 99us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:48:44.882213 (+ 529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:48:44.882227 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:48:44.883241 (+ 1014us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:48:44.883244 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:48:44.884324 (+ 1080us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:48:44.884337 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:48:44.884898 (+ 561us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:48:44.884903 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:48:44.885088 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:48:44.885839 (+ 751us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 22:48:44.885841 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:48:44.885889 (+ 48us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 22:48:44.885965 (+ 76us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:48:44.886303 (+ 338us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:48:44.886316 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:48:44.886329 (+ 13us) client_negotiation.cc:769] Sending connection context
0925 22:48:44.886466 (+ 137us) client_negotiation.cc:240] Negotiation successful
0925 22:48:44.886812 (+ 346us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":302,"thread_start_us":136,"threads_started":1}
I20260925 22:48:44.889109 30952 heartbeater.cc:347] Connected to a master server at 127.10.198.62:39021
I20260925 22:48:44.889497 30952 heartbeater.cc:464] Registering TS with master...
I20260925 22:48:44.890298 30952 heartbeater.cc:511] Master 127.10.198.62:39021 requested a full tablet report, sending...
I20260925 22:48:44.892441 30764 ts_manager.cc:194] Registered new tserver with Master: 40d68243fa5344f78b05bd14d8516a5e (127.10.198.1:44215)
I20260925 22:48:44.894420 30764 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.10.198.1:60453
W20260925 22:48:45.004179 30955 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:48:45.004505 30955 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:48:45.004547 30955 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:48:45.009059 30955 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:48:45.009143 30955 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:48:45.009205 30955 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.198.2
I20260925 22:48:45.014394 30955 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.198.20:36321
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790376259650642-11032-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790376259650642-11032-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.198.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790376259650642-11032-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.198.2
--webserver_port=0
--tserver_master_addrs=127.10.198.62:39021
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30955
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.198.2
--log_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790376259650642-11032-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:09:16 UTC on ccycloud.quasar-afmhyw.root.comops.site
build id 4673379
I20260925 22:48:45.015646 30955 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:48:45.017118 30955 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 22:48:45.027587 30955 server_base.cc:1034] running on GCE node
W20260925 22:48:45.027554 30960 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:48:45.027524 30961 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:48:45.027526 30963 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:48:45.028281 30955 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:48:45.029145 30955 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:48:45.030349 30955 hybrid_clock.cc:648] HybridClock initialized: now 1790376525030313 us; error 55 us; skew 500 ppm
I20260925 22:48:45.032541 30955 webserver.cc:460] Webserver started at http://127.10.198.2:41443/ using document root <none> and password file <none>
I20260925 22:48:45.033209 30955 fs_manager.cc:359] Metadata directory not provided
I20260925 22:48:45.033325 30955 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:48:45.033604 30955 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:48:45.035954 30955 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790376259650642-11032-0/minicluster-data/ts-1/data/instance:
uuid: "42d4fd97491841348de7dae820f1013b"
format_stamp: "Formatted at 2026-09-25 22:48:45 on dist-test-slave-5vdr"
server_key: "86f6de57e302faa347e282409d6975f6"
server_key_iv: "01394a3963e4c2a157d395e78ecc6bb7"
server_key_version: "encryptionkey@0"
I20260925 22:48:45.036617 30955 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790376259650642-11032-0/minicluster-data/ts-1/wal/instance:
uuid: "42d4fd97491841348de7dae820f1013b"
format_stamp: "Formatted at 2026-09-25 22:48:45 on dist-test-slave-5vdr"
server_key: "86f6de57e302faa347e282409d6975f6"
server_key_iv: "01394a3963e4c2a157d395e78ecc6bb7"
server_key_version: "encryptionkey@0"
I20260925 22:48:45.040920 30955 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260925 22:48:45.043771 30969 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:48:45.045285 30955 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260925 22:48:45.045464 30955 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790376259650642-11032-0/minicluster-data/ts-1/data,/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790376259650642-11032-0/minicluster-data/ts-1/wal
uuid: "42d4fd97491841348de7dae820f1013b"
format_stamp: "Formatted at 2026-09-25 22:48:45 on dist-test-slave-5vdr"
server_key: "86f6de57e302faa347e282409d6975f6"
server_key_iv: "01394a3963e4c2a157d395e78ecc6bb7"
server_key_version: "encryptionkey@0"
I20260925 22:48:45.045621 30955 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790376259650642-11032-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790376259650642-11032-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790376259650642-11032-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:48:45.057960 30955 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:48:45.058763 30955 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:48:45.059000 30955 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:48:45.059664 30955 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 22:48:45.060927 30955 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:48:45.061012 30955 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:48:45.061093 30955 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:48:45.061146 30955 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 22:48:45.070709 30955 rpc_server.cc:307] RPC server started. Bound to: 127.10.198.2:41859
I20260925 22:48:45.070725 31082 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.198.2:41859 every 8 connection(s)
I20260925 22:48:45.071730 30955 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790376259650642-11032-0/minicluster-data/ts-1/data/info.pb
I20260925 22:48:45.072124 11032 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrWXm8X/build/debug/bin/kudu as pid 30955
I20260925 22:48:45.072254 11032 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790376259650642-11032-0/minicluster-data/ts-1/wal/instance
I20260925 22:48:45.072623 11032 external_mini_cluster.cc:1428] Setting key acdcf47dc928d0896dc8a86ab7435fdc
I20260925 22:48:45.075080 11032 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrWXm8X/build/debug/bin/kudu
/tmp/dist-test-taskrWXm8X/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790376259650642-11032-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790376259650642-11032-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790376259650642-11032-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790376259650642-11032-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.198.3:0
--local_ip_for_outbound_sockets=127.10.198.3
--webserver_interface=127.10.198.3
--webserver_port=0
--tserver_master_addrs=127.10.198.62:39021
--builtin_ntp_servers=127.10.198.20:36321
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260925 22:48:45.083787 30803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:48:45.074080 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.198.2:43607 (local address 127.10.198.62:39021)
0925 22:48:45.074229 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0925 22:48:45.074233 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:48:45.075753 (+ 1520us) server_negotiation.cc:407] Connection header received
0925 22:48:45.077614 (+ 1861us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:48:45.077618 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:48:45.077680 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:48:45.077772 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:48:45.078843 (+ 1071us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:48:45.079790 (+ 947us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:48:45.081104 (+ 1314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:48:45.081551 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:48:45.081698 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:48:45.083068 (+ 1370us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:48:45.083076 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:48:45.083080 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 22:48:45.083100 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:48:45.083130 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 22:48:45.083143 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:48:45.083244 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0925 22:48:45.083432 (+ 188us) server_negotiation.cc:299] Negotiation successful
0925 22:48:45.083538 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260925 22:48:45.084899 31085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:48:45.074679 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.198.62:39021 (local address 127.10.198.2:43607)
0925 22:48:45.075557 (+ 878us) negotiation.cc:107] Waiting for socket to connect
0925 22:48:45.075621 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0925 22:48:45.077360 (+ 1739us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:48:45.077970 (+ 610us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:48:45.077978 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:48:45.078094 (+ 116us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:48:45.078624 (+ 530us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:48:45.078669 (+ 45us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:48:45.079971 (+ 1302us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:48:45.079974 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:48:45.080946 (+ 972us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:48:45.080957 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:48:45.081748 (+ 791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:48:45.081751 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:48:45.081922 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:48:45.082725 (+ 803us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 22:48:45.082726 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:48:45.082778 (+ 52us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 22:48:45.082860 (+ 82us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:48:45.083254 (+ 394us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:48:45.083263 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:48:45.083268 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 22:48:45.083396 (+ 128us) client_negotiation.cc:240] Negotiation successful
0925 22:48:45.083675 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":329,"thread_start_us":131,"threads_started":1}
I20260925 22:48:45.086362 31083 heartbeater.cc:347] Connected to a master server at 127.10.198.62:39021
I20260925 22:48:45.086690 31083 heartbeater.cc:464] Registering TS with master...
I20260925 22:48:45.087386 31083 heartbeater.cc:511] Master 127.10.198.62:39021 requested a full tablet report, sending...
I20260925 22:48:45.088747 30764 ts_manager.cc:194] Registered new tserver with Master: 42d4fd97491841348de7dae820f1013b (127.10.198.2:41859)
I20260925 22:48:45.089476 30764 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.10.198.2:43607
W20260925 22:48:45.212647 31086 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:48:45.212999 31086 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:48:45.213091 31086 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:48:45.220494 31086 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:48:45.220618 31086 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:48:45.220717 31086 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.198.3
I20260925 22:48:45.226670 31086 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.198.20:36321
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790376259650642-11032-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790376259650642-11032-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.198.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790376259650642-11032-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.198.3
--webserver_port=0
--tserver_master_addrs=127.10.198.62:39021
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31086
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.198.3
--log_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790376259650642-11032-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:09:16 UTC on ccycloud.quasar-afmhyw.root.comops.site
build id 4673379
I20260925 22:48:45.228035 31086 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:48:45.229615 31086 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:48:45.240469 31094 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:48:45.240554 31091 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:48:45.240530 31086 server_base.cc:1034] running on GCE node
W20260925 22:48:45.240474 31092 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:48:45.241293 31086 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:48:45.242117 31086 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:48:45.243345 31086 hybrid_clock.cc:648] HybridClock initialized: now 1790376525243306 us; error 55 us; skew 500 ppm
I20260925 22:48:45.245671 31086 webserver.cc:460] Webserver started at http://127.10.198.3:36545/ using document root <none> and password file <none>
I20260925 22:48:45.246353 31086 fs_manager.cc:359] Metadata directory not provided
I20260925 22:48:45.246428 31086 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:48:45.246686 31086 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:48:45.249330 31086 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790376259650642-11032-0/minicluster-data/ts-2/data/instance:
uuid: "03650d60099948d9b1464bb000859dc9"
format_stamp: "Formatted at 2026-09-25 22:48:45 on dist-test-slave-5vdr"
server_key: "b811548a5fcc23b05c71a6559d2fb53c"
server_key_iv: "dcacf0f00321c88a3ac1e71c886160e1"
server_key_version: "encryptionkey@0"
I20260925 22:48:45.249972 31086 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790376259650642-11032-0/minicluster-data/ts-2/wal/instance:
uuid: "03650d60099948d9b1464bb000859dc9"
format_stamp: "Formatted at 2026-09-25 22:48:45 on dist-test-slave-5vdr"
server_key: "b811548a5fcc23b05c71a6559d2fb53c"
server_key_iv: "dcacf0f00321c88a3ac1e71c886160e1"
server_key_version: "encryptionkey@0"
I20260925 22:48:45.254506 31086 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260925 22:48:45.257388 31100 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:48:45.258855 31086 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260925 22:48:45.258988 31086 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790376259650642-11032-0/minicluster-data/ts-2/data,/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790376259650642-11032-0/minicluster-data/ts-2/wal
uuid: "03650d60099948d9b1464bb000859dc9"
format_stamp: "Formatted at 2026-09-25 22:48:45 on dist-test-slave-5vdr"
server_key: "b811548a5fcc23b05c71a6559d2fb53c"
server_key_iv: "dcacf0f00321c88a3ac1e71c886160e1"
server_key_version: "encryptionkey@0"
I20260925 22:48:45.259191 31086 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790376259650642-11032-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790376259650642-11032-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790376259650642-11032-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:48:45.285019 31086 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:48:45.285835 31086 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:48:45.286070 31086 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:48:45.286725 31086 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 22:48:45.287752 31086 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 22:48:45.287809 31086 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 22:48:45.287910 31086 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 22:48:45.287949 31086 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 22:48:45.297781 31086 rpc_server.cc:307] RPC server started. Bound to: 127.10.198.3:42937
I20260925 22:48:45.297799 31213 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.198.3:42937 every 8 connection(s)
I20260925 22:48:45.298839 31086 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790376259650642-11032-0/minicluster-data/ts-2/data/info.pb
I20260925 22:48:45.303448 11032 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrWXm8X/build/debug/bin/kudu as pid 31086
I20260925 22:48:45.303619 11032 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790376259650642-11032-0/minicluster-data/ts-2/wal/instance
I20260925 22:48:45.304105 11032 external_mini_cluster.cc:1428] Setting key 923b7ea075e6099a765b8c7fb7059f16
I20260925 22:48:45.310014 30803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:48:45.300923 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.198.3:54319 (local address 127.10.198.62:39021)
0925 22:48:45.301074 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0925 22:48:45.301078 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:48:45.302289 (+ 1211us) server_negotiation.cc:407] Connection header received
0925 22:48:45.303909 (+ 1620us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:48:45.303914 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:48:45.303993 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:48:45.304117 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:48:45.305458 (+ 1341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:48:45.306142 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:48:45.307591 (+ 1449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:48:45.307969 (+ 378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:48:45.308078 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:48:45.309456 (+ 1378us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:48:45.309459 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:48:45.309462 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 22:48:45.309495 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:48:45.309533 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 22:48:45.309544 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:48:45.309621 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0925 22:48:45.309803 (+ 182us) server_negotiation.cc:299] Negotiation successful
0925 22:48:45.309872 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260925 22:48:45.311086 31216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:48:45.301326 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.198.62:39021 (local address 127.10.198.3:54319)
0925 22:48:45.302109 (+ 783us) negotiation.cc:107] Waiting for socket to connect
0925 22:48:45.302157 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0925 22:48:45.303479 (+ 1322us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:48:45.304351 (+ 872us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:48:45.304360 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:48:45.304462 (+ 102us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:48:45.305260 (+ 798us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:48:45.305284 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:48:45.306297 (+ 1013us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:48:45.306300 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:48:45.307444 (+ 1144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:48:45.307456 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:48:45.308177 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:48:45.308180 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:48:45.308368 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:48:45.309135 (+ 767us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 22:48:45.309136 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:48:45.309185 (+ 49us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 22:48:45.309265 (+ 80us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:48:45.309644 (+ 379us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:48:45.309653 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:48:45.309658 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 22:48:45.309785 (+ 127us) client_negotiation.cc:240] Negotiation successful
0925 22:48:45.310112 (+ 327us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":350,"thread_start_us":169,"threads_started":1}
I20260925 22:48:45.312577 31214 heartbeater.cc:347] Connected to a master server at 127.10.198.62:39021
I20260925 22:48:45.312935 31214 heartbeater.cc:464] Registering TS with master...
I20260925 22:48:45.313690 31214 heartbeater.cc:511] Master 127.10.198.62:39021 requested a full tablet report, sending...
I20260925 22:48:45.315090 30764 ts_manager.cc:194] Registered new tserver with Master: 03650d60099948d9b1464bb000859dc9 (127.10.198.3:42937)
I20260925 22:48:45.315688 30764 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.10.198.3:54319
I20260925 22:48:45.318678 11032 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 22:48:45.324714 30803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:48:45.320769 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49320 (local address 127.10.198.62:39021)
0925 22:48:45.320942 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0925 22:48:45.320947 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:48:45.321090 (+ 143us) server_negotiation.cc:407] Connection header received
0925 22:48:45.321165 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:48:45.321167 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:48:45.321213 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:48:45.321292 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:48:45.321555 (+ 263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:48:45.322083 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:48:45.322950 (+ 867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:48:45.323231 (+ 281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:48:45.323326 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:48:45.324254 (+ 928us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:48:45.324258 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:48:45.324260 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 22:48:45.324273 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:48:45.324293 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 22:48:45.324300 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:48:45.324362 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0925 22:48:45.324458 (+ 96us) server_negotiation.cc:299] Negotiation successful
0925 22:48:45.324520 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260925 22:48:45.328353 30764 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49320:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 22:48:45.331748 30764 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 22:48:45.348704 31230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:48:45.342657 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41868 (local address 127.10.198.2:41859)
0925 22:48:45.343440 (+ 783us) server_negotiation.cc:206] Beginning negotiation
0925 22:48:45.343456 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0925 22:48:45.343498 (+ 42us) server_negotiation.cc:407] Connection header received
0925 22:48:45.343586 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:48:45.343594 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:48:45.343816 (+ 222us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:48:45.344011 (+ 195us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:48:45.344502 (+ 491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:48:45.345281 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:48:45.347078 (+ 1797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:48:45.347774 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:48:45.347997 (+ 223us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:48:45.348252 (+ 255us) server_negotiation.cc:1035] Waiting for connection context
0925 22:48:45.348384 (+ 132us) server_negotiation.cc:299] Negotiation successful
0925 22:48:45.348505 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":667,"thread_start_us":128,"threads_started":1}
I20260925 22:48:45.348701 31228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:48:45.342026 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.198.2:41859 (local address 127.0.0.1:41868)
0925 22:48:45.342815 (+ 789us) negotiation.cc:107] Waiting for socket to connect
0925 22:48:45.342879 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0925 22:48:45.343028 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:48:45.343970 (+ 942us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:48:45.343972 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:48:45.343983 (+ 11us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:48:45.344394 (+ 411us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:48:45.344398 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:48:45.345421 (+ 1023us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:48:45.345426 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:48:45.346912 (+ 1486us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:48:45.346925 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:48:45.348006 (+ 1081us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:48:45.348011 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:48:45.348156 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:48:45.348203 (+ 47us) client_negotiation.cc:769] Sending connection context
0925 22:48:45.348350 (+ 147us) client_negotiation.cc:240] Negotiation successful
0925 22:48:45.348515 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":693,"mutex_wait_us":267,"thread_start_us":120,"threads_started":1}
I20260925 22:48:45.349141 31227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:48:45.341775 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.198.1:44215 (local address 127.0.0.1:40794)
0925 22:48:45.342817 (+ 1042us) negotiation.cc:107] Waiting for socket to connect
0925 22:48:45.342880 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0925 22:48:45.343081 (+ 201us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:48:45.343933 (+ 852us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:48:45.343941 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:48:45.343969 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:48:45.344200 (+ 231us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:48:45.344209 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:48:45.345785 (+ 1576us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:48:45.345789 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:48:45.346981 (+ 1192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:48:45.346992 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:48:45.348430 (+ 1438us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:48:45.348434 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:48:45.348643 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:48:45.348682 (+ 39us) client_negotiation.cc:769] Sending connection context
0925 22:48:45.348744 (+ 62us) client_negotiation.cc:240] Negotiation successful
0925 22:48:45.348995 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":835,"mutex_wait_us":183,"thread_start_us":111,"threads_started":1}
I20260925 22:48:45.349145 31229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:48:45.342677 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41966 (local address 127.10.198.3:42937)
0925 22:48:45.343087 (+ 410us) server_negotiation.cc:206] Beginning negotiation
0925 22:48:45.343102 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0925 22:48:45.343624 (+ 522us) server_negotiation.cc:407] Connection header received
0925 22:48:45.343706 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:48:45.343711 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:48:45.343925 (+ 214us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:48:45.344071 (+ 146us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:48:45.344381 (+ 310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:48:45.345647 (+ 1266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:48:45.347214 (+ 1567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:48:45.348051 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:48:45.348169 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:48:45.348361 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0925 22:48:45.348834 (+ 473us) server_negotiation.cc:299] Negotiation successful
0925 22:48:45.348962 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":303,"thread_start_us":86,"threads_started":1}
I20260925 22:48:45.349843 31226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:48:45.342023 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40794 (local address 127.10.198.1:44215)
0925 22:48:45.342630 (+ 607us) server_negotiation.cc:206] Beginning negotiation
0925 22:48:45.342637 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 22:48:45.342985 (+ 348us) server_negotiation.cc:407] Connection header received
0925 22:48:45.343320 (+ 335us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:48:45.343327 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:48:45.343573 (+ 246us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:48:45.343754 (+ 181us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:48:45.344372 (+ 618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:48:45.345651 (+ 1279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:48:45.347108 (+ 1457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:48:45.348065 (+ 957us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:48:45.348169 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:48:45.348361 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0925 22:48:45.349565 (+ 1204us) server_negotiation.cc:299] Negotiation successful
0925 22:48:45.349667 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":468,"thread_start_us":111,"threads_started":1}
I20260925 22:48:45.350229 31225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:48:45.342282 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.198.3:42937 (local address 127.0.0.1:41966)
0925 22:48:45.343396 (+ 1114us) negotiation.cc:107] Waiting for socket to connect
0925 22:48:45.343412 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 22:48:45.343509 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:48:45.344136 (+ 627us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:48:45.344139 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:48:45.344151 (+ 12us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:48:45.344259 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:48:45.344267 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:48:45.345837 (+ 1570us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:48:45.345842 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:48:45.347059 (+ 1217us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:48:45.347073 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:48:45.348430 (+ 1357us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:48:45.348434 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:48:45.348571 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:48:45.348666 (+ 95us) client_negotiation.cc:769] Sending connection context
0925 22:48:45.348744 (+ 78us) client_negotiation.cc:240] Negotiation successful
0925 22:48:45.350107 (+ 1363us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1053,"mutex_wait_us":1339,"thread_start_us":90,"threads_started":1}
I20260925 22:48:45.352550 31017 tablet_service.cc:1467] Processing CreateTablet for tablet 18f5931027914ecb954e474b9d46b1ac (DEFAULT_TABLE table=test-table [id=e746ba589d2c443a80decfcdfc4ff7e9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 22:48:45.353232 31148 tablet_service.cc:1467] Processing CreateTablet for tablet 18f5931027914ecb954e474b9d46b1ac (DEFAULT_TABLE table=test-table [id=e746ba589d2c443a80decfcdfc4ff7e9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 22:48:45.353749 31017 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 18f5931027914ecb954e474b9d46b1ac. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:48:45.354231 31148 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 18f5931027914ecb954e474b9d46b1ac. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:48:45.361754 31232 tablet_bootstrap.cc:492] T 18f5931027914ecb954e474b9d46b1ac P 03650d60099948d9b1464bb000859dc9: Bootstrap starting.
I20260925 22:48:45.362943 30886 tablet_service.cc:1467] Processing CreateTablet for tablet 18f5931027914ecb954e474b9d46b1ac (DEFAULT_TABLE table=test-table [id=e746ba589d2c443a80decfcdfc4ff7e9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 22:48:45.364003 31231 tablet_bootstrap.cc:492] T 18f5931027914ecb954e474b9d46b1ac P 42d4fd97491841348de7dae820f1013b: Bootstrap starting.
I20260925 22:48:45.364169 30886 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 18f5931027914ecb954e474b9d46b1ac. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:48:45.366197 31231 tablet_bootstrap.cc:654] T 18f5931027914ecb954e474b9d46b1ac P 42d4fd97491841348de7dae820f1013b: Neither blocks nor log segments found. Creating new log.
I20260925 22:48:45.367129 31232 tablet_bootstrap.cc:654] T 18f5931027914ecb954e474b9d46b1ac P 03650d60099948d9b1464bb000859dc9: Neither blocks nor log segments found. Creating new log.
I20260925 22:48:45.367216 31231 log.cc:824] T 18f5931027914ecb954e474b9d46b1ac P 42d4fd97491841348de7dae820f1013b: Log is configured to *not* fsync() on all Append() calls
I20260925 22:48:45.368252 31232 log.cc:824] T 18f5931027914ecb954e474b9d46b1ac P 03650d60099948d9b1464bb000859dc9: Log is configured to *not* fsync() on all Append() calls
I20260925 22:48:45.369676 31231 tablet_bootstrap.cc:492] T 18f5931027914ecb954e474b9d46b1ac P 42d4fd97491841348de7dae820f1013b: No bootstrap required, opened a new log
I20260925 22:48:45.369922 31231 ts_tablet_manager.cc:1397] T 18f5931027914ecb954e474b9d46b1ac P 42d4fd97491841348de7dae820f1013b: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260925 22:48:45.370501 31232 tablet_bootstrap.cc:492] T 18f5931027914ecb954e474b9d46b1ac P 03650d60099948d9b1464bb000859dc9: No bootstrap required, opened a new log
I20260925 22:48:45.370712 31232 ts_tablet_manager.cc:1397] T 18f5931027914ecb954e474b9d46b1ac P 03650d60099948d9b1464bb000859dc9: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.000s
I20260925 22:48:45.371665 31235 tablet_bootstrap.cc:492] T 18f5931027914ecb954e474b9d46b1ac P 40d68243fa5344f78b05bd14d8516a5e: Bootstrap starting.
I20260925 22:48:45.374033 31235 tablet_bootstrap.cc:654] T 18f5931027914ecb954e474b9d46b1ac P 40d68243fa5344f78b05bd14d8516a5e: Neither blocks nor log segments found. Creating new log.
I20260925 22:48:45.374997 31235 log.cc:824] T 18f5931027914ecb954e474b9d46b1ac P 40d68243fa5344f78b05bd14d8516a5e: Log is configured to *not* fsync() on all Append() calls
I20260925 22:48:45.375317 31232 raft_consensus.cc:348] T 18f5931027914ecb954e474b9d46b1ac P 03650d60099948d9b1464bb000859dc9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40d68243fa5344f78b05bd14d8516a5e" member_type: VOTER last_known_addr { host: "127.10.198.1" port: 44215 } } peers { permanent_uuid: "42d4fd97491841348de7dae820f1013b" member_type: VOTER last_known_addr { host: "127.10.198.2" port: 41859 } } peers { permanent_uuid: "03650d60099948d9b1464bb000859dc9" member_type: VOTER last_known_addr { host: "127.10.198.3" port: 42937 } }
I20260925 22:48:45.375317 31231 raft_consensus.cc:348] T 18f5931027914ecb954e474b9d46b1ac P 42d4fd97491841348de7dae820f1013b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40d68243fa5344f78b05bd14d8516a5e" member_type: VOTER last_known_addr { host: "127.10.198.1" port: 44215 } } peers { permanent_uuid: "42d4fd97491841348de7dae820f1013b" member_type: VOTER last_known_addr { host: "127.10.198.2" port: 41859 } } peers { permanent_uuid: "03650d60099948d9b1464bb000859dc9" member_type: VOTER last_known_addr { host: "127.10.198.3" port: 42937 } }
I20260925 22:48:45.375633 31232 raft_consensus.cc:374] T 18f5931027914ecb954e474b9d46b1ac P 03650d60099948d9b1464bb000859dc9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:48:45.375633 31231 raft_consensus.cc:374] T 18f5931027914ecb954e474b9d46b1ac P 42d4fd97491841348de7dae820f1013b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:48:45.375718 31231 raft_consensus.cc:729] T 18f5931027914ecb954e474b9d46b1ac P 42d4fd97491841348de7dae820f1013b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 42d4fd97491841348de7dae820f1013b, State: Initialized, Role: FOLLOWER
I20260925 22:48:45.375718 31232 raft_consensus.cc:729] T 18f5931027914ecb954e474b9d46b1ac P 03650d60099948d9b1464bb000859dc9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 03650d60099948d9b1464bb000859dc9, State: Initialized, Role: FOLLOWER
I20260925 22:48:45.376297 31231 consensus_queue.cc:260] T 18f5931027914ecb954e474b9d46b1ac P 42d4fd97491841348de7dae820f1013b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40d68243fa5344f78b05bd14d8516a5e" member_type: VOTER last_known_addr { host: "127.10.198.1" port: 44215 } } peers { permanent_uuid: "42d4fd97491841348de7dae820f1013b" member_type: VOTER last_known_addr { host: "127.10.198.2" port: 41859 } } peers { permanent_uuid: "03650d60099948d9b1464bb000859dc9" member_type: VOTER last_known_addr { host: "127.10.198.3" port: 42937 } }
I20260925 22:48:45.376297 31232 consensus_queue.cc:260] T 18f5931027914ecb954e474b9d46b1ac P 03650d60099948d9b1464bb000859dc9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40d68243fa5344f78b05bd14d8516a5e" member_type: VOTER last_known_addr { host: "127.10.198.1" port: 44215 } } peers { permanent_uuid: "42d4fd97491841348de7dae820f1013b" member_type: VOTER last_known_addr { host: "127.10.198.2" port: 41859 } } peers { permanent_uuid: "03650d60099948d9b1464bb000859dc9" member_type: VOTER last_known_addr { host: "127.10.198.3" port: 42937 } }
I20260925 22:48:45.377305 31083 heartbeater.cc:503] Master 127.10.198.62:39021 was elected leader, sending a full tablet report...
I20260925 22:48:45.377334 31214 heartbeater.cc:503] Master 127.10.198.62:39021 was elected leader, sending a full tablet report...
I20260925 22:48:45.377449 31231 ts_tablet_manager.cc:1428] T 18f5931027914ecb954e474b9d46b1ac P 42d4fd97491841348de7dae820f1013b: Time spent starting tablet: real 0.007s user 0.008s sys 0.000s
I20260925 22:48:45.378028 31235 tablet_bootstrap.cc:492] T 18f5931027914ecb954e474b9d46b1ac P 40d68243fa5344f78b05bd14d8516a5e: No bootstrap required, opened a new log
I20260925 22:48:45.378073 31232 ts_tablet_manager.cc:1428] T 18f5931027914ecb954e474b9d46b1ac P 03650d60099948d9b1464bb000859dc9: Time spent starting tablet: real 0.007s user 0.008s sys 0.000s
I20260925 22:48:45.378464 31235 ts_tablet_manager.cc:1397] T 18f5931027914ecb954e474b9d46b1ac P 40d68243fa5344f78b05bd14d8516a5e: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20260925 22:48:45.383260 31235 raft_consensus.cc:348] T 18f5931027914ecb954e474b9d46b1ac P 40d68243fa5344f78b05bd14d8516a5e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40d68243fa5344f78b05bd14d8516a5e" member_type: VOTER last_known_addr { host: "127.10.198.1" port: 44215 } } peers { permanent_uuid: "42d4fd97491841348de7dae820f1013b" member_type: VOTER last_known_addr { host: "127.10.198.2" port: 41859 } } peers { permanent_uuid: "03650d60099948d9b1464bb000859dc9" member_type: VOTER last_known_addr { host: "127.10.198.3" port: 42937 } }
I20260925 22:48:45.383540 31235 raft_consensus.cc:374] T 18f5931027914ecb954e474b9d46b1ac P 40d68243fa5344f78b05bd14d8516a5e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:48:45.383576 31235 raft_consensus.cc:729] T 18f5931027914ecb954e474b9d46b1ac P 40d68243fa5344f78b05bd14d8516a5e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 40d68243fa5344f78b05bd14d8516a5e, State: Initialized, Role: FOLLOWER
I20260925 22:48:45.384073 31235 consensus_queue.cc:260] T 18f5931027914ecb954e474b9d46b1ac P 40d68243fa5344f78b05bd14d8516a5e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40d68243fa5344f78b05bd14d8516a5e" member_type: VOTER last_known_addr { host: "127.10.198.1" port: 44215 } } peers { permanent_uuid: "42d4fd97491841348de7dae820f1013b" member_type: VOTER last_known_addr { host: "127.10.198.2" port: 41859 } } peers { permanent_uuid: "03650d60099948d9b1464bb000859dc9" member_type: VOTER last_known_addr { host: "127.10.198.3" port: 42937 } }
I20260925 22:48:45.384812 30952 heartbeater.cc:503] Master 127.10.198.62:39021 was elected leader, sending a full tablet report...
I20260925 22:48:45.385046 31235 ts_tablet_manager.cc:1428] T 18f5931027914ecb954e474b9d46b1ac P 40d68243fa5344f78b05bd14d8516a5e: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260925 22:48:45.549655 31239 raft_consensus.cc:482] T 18f5931027914ecb954e474b9d46b1ac P 40d68243fa5344f78b05bd14d8516a5e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260925 22:48:45.549964 31215 tablet.cc:2369] T 18f5931027914ecb954e474b9d46b1ac P 03650d60099948d9b1464bb000859dc9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 22:48:45.550009 31239 raft_consensus.cc:504] T 18f5931027914ecb954e474b9d46b1ac P 40d68243fa5344f78b05bd14d8516a5e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40d68243fa5344f78b05bd14d8516a5e" member_type: VOTER last_known_addr { host: "127.10.198.1" port: 44215 } } peers { permanent_uuid: "42d4fd97491841348de7dae820f1013b" member_type: VOTER last_known_addr { host: "127.10.198.2" port: 41859 } } peers { permanent_uuid: "03650d60099948d9b1464bb000859dc9" member_type: VOTER last_known_addr { host: "127.10.198.3" port: 42937 } }
I20260925 22:48:45.551246 31239 leader_election.cc:288] T 18f5931027914ecb954e474b9d46b1ac P 40d68243fa5344f78b05bd14d8516a5e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 42d4fd97491841348de7dae820f1013b (127.10.198.2:41859), 03650d60099948d9b1464bb000859dc9 (127.10.198.3:42937)
I20260925 22:48:45.555702 31229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:48:45.551655 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.198.1:60269 (local address 127.10.198.3:42937)
0925 22:48:45.551803 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0925 22:48:45.551809 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 22:48:45.552081 (+ 272us) server_negotiation.cc:407] Connection header received
0925 22:48:45.552215 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:48:45.552218 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:48:45.552280 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:48:45.552363 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:48:45.552738 (+ 375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:48:45.553376 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:48:45.554552 (+ 1176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:48:45.555045 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:48:45.555150 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:48:45.555232 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0925 22:48:45.555447 (+ 215us) server_negotiation.cc:299] Negotiation successful
0925 22:48:45.555538 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260925 22:48:45.555724 31230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:48:45.551724 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.198.1:56085 (local address 127.10.198.2:41859)
0925 22:48:45.551847 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0925 22:48:45.551875 (+ 28us) server_negotiation.cc:399] Waiting for connection header
0925 22:48:45.552189 (+ 314us) server_negotiation.cc:407] Connection header received
0925 22:48:45.552366 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:48:45.552370 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:48:45.552429 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:48:45.552526 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 22:48:45.552932 (+ 406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:48:45.553475 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:48:45.554691 (+ 1216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:48:45.555082 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:48:45.555242 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:48:45.555297 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0925 22:48:45.555506 (+ 209us) server_negotiation.cc:299] Negotiation successful
0925 22:48:45.555586 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260925 22:48:45.555737 31241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:48:45.551512 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.198.3:42937 (local address 127.10.198.1:60269)
0925 22:48:45.551968 (+ 456us) negotiation.cc:107] Waiting for socket to connect
0925 22:48:45.551981 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 22:48:45.552093 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:48:45.552393 (+ 300us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:48:45.552396 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:48:45.552412 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:48:45.552601 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:48:45.552610 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:48:45.553502 (+ 892us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:48:45.553508 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:48:45.554412 (+ 904us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:48:45.554423 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:48:45.555180 (+ 757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:48:45.555184 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:48:45.555298 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:48:45.555331 (+ 33us) client_negotiation.cc:769] Sending connection context
0925 22:48:45.555405 (+ 74us) client_negotiation.cc:240] Negotiation successful
0925 22:48:45.555546 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":382,"mutex_wait_us":27,"thread_start_us":108,"threads_started":1}
I20260925 22:48:45.555733 31240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:48:45.551511 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.198.2:41859 (local address 127.10.198.1:56085)
0925 22:48:45.552058 (+ 547us) negotiation.cc:107] Waiting for socket to connect
0925 22:48:45.552077 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0925 22:48:45.552204 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:48:45.552567 (+ 363us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:48:45.552572 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:48:45.552592 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 22:48:45.552739 (+ 147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:48:45.552747 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:48:45.553607 (+ 860us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:48:45.553610 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:48:45.554554 (+ 944us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:48:45.554564 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:48:45.555243 (+ 679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:48:45.555250 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:48:45.555350 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:48:45.555367 (+ 17us) client_negotiation.cc:769] Sending connection context
0925 22:48:45.555466 (+ 99us) client_negotiation.cc:240] Negotiation successful
0925 22:48:45.555557 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":461,"mutex_wait_us":75,"thread_start_us":90,"threads_started":1}
I20260925 22:48:45.556638 31168 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18f5931027914ecb954e474b9d46b1ac" candidate_uuid: "40d68243fa5344f78b05bd14d8516a5e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "03650d60099948d9b1464bb000859dc9" is_pre_election: true
I20260925 22:48:45.556814 31037 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18f5931027914ecb954e474b9d46b1ac" candidate_uuid: "40d68243fa5344f78b05bd14d8516a5e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "42d4fd97491841348de7dae820f1013b" is_pre_election: true
I20260925 22:48:45.557052 31037 raft_consensus.cc:2454] T 18f5931027914ecb954e474b9d46b1ac P 42d4fd97491841348de7dae820f1013b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 40d68243fa5344f78b05bd14d8516a5e in term 0.
I20260925 22:48:45.557036 31168 raft_consensus.cc:2454] T 18f5931027914ecb954e474b9d46b1ac P 03650d60099948d9b1464bb000859dc9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 40d68243fa5344f78b05bd14d8516a5e in term 0.
I20260925 22:48:45.557603 30841 leader_election.cc:302] T 18f5931027914ecb954e474b9d46b1ac P 40d68243fa5344f78b05bd14d8516a5e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 03650d60099948d9b1464bb000859dc9, 40d68243fa5344f78b05bd14d8516a5e, 42d4fd97491841348de7dae820f1013b; no voters:
I20260925 22:48:45.557916 31239 raft_consensus.cc:2781] T 18f5931027914ecb954e474b9d46b1ac P 40d68243fa5344f78b05bd14d8516a5e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 22:48:45.558009 31239 raft_consensus.cc:482] T 18f5931027914ecb954e474b9d46b1ac P 40d68243fa5344f78b05bd14d8516a5e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 22:48:45.558040 31239 raft_consensus.cc:3037] T 18f5931027914ecb954e474b9d46b1ac P 40d68243fa5344f78b05bd14d8516a5e [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:48:45.559158 31239 raft_consensus.cc:504] T 18f5931027914ecb954e474b9d46b1ac P 40d68243fa5344f78b05bd14d8516a5e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40d68243fa5344f78b05bd14d8516a5e" member_type: VOTER last_known_addr { host: "127.10.198.1" port: 44215 } } peers { permanent_uuid: "42d4fd97491841348de7dae820f1013b" member_type: VOTER last_known_addr { host: "127.10.198.2" port: 41859 } } peers { permanent_uuid: "03650d60099948d9b1464bb000859dc9" member_type: VOTER last_known_addr { host: "127.10.198.3" port: 42937 } }
I20260925 22:48:45.559562 31239 leader_election.cc:288] T 18f5931027914ecb954e474b9d46b1ac P 40d68243fa5344f78b05bd14d8516a5e [CANDIDATE]: Term 1 election: Requested vote from peers 42d4fd97491841348de7dae820f1013b (127.10.198.2:41859), 03650d60099948d9b1464bb000859dc9 (127.10.198.3:42937)
I20260925 22:48:45.560076 31037 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18f5931027914ecb954e474b9d46b1ac" candidate_uuid: "40d68243fa5344f78b05bd14d8516a5e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "42d4fd97491841348de7dae820f1013b"
I20260925 22:48:45.560076 31168 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18f5931027914ecb954e474b9d46b1ac" candidate_uuid: "40d68243fa5344f78b05bd14d8516a5e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "03650d60099948d9b1464bb000859dc9"
I20260925 22:48:45.560231 31168 raft_consensus.cc:3037] T 18f5931027914ecb954e474b9d46b1ac P 03650d60099948d9b1464bb000859dc9 [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:48:45.560231 31037 raft_consensus.cc:3037] T 18f5931027914ecb954e474b9d46b1ac P 42d4fd97491841348de7dae820f1013b [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:48:45.561609 31168 raft_consensus.cc:2454] T 18f5931027914ecb954e474b9d46b1ac P 03650d60099948d9b1464bb000859dc9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 40d68243fa5344f78b05bd14d8516a5e in term 1.
I20260925 22:48:45.561609 31037 raft_consensus.cc:2454] T 18f5931027914ecb954e474b9d46b1ac P 42d4fd97491841348de7dae820f1013b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 40d68243fa5344f78b05bd14d8516a5e in term 1.
I20260925 22:48:45.562049 30840 leader_election.cc:302] T 18f5931027914ecb954e474b9d46b1ac P 40d68243fa5344f78b05bd14d8516a5e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 03650d60099948d9b1464bb000859dc9, 40d68243fa5344f78b05bd14d8516a5e, 42d4fd97491841348de7dae820f1013b; no voters:
I20260925 22:48:45.562243 31239 raft_consensus.cc:2781] T 18f5931027914ecb954e474b9d46b1ac P 40d68243fa5344f78b05bd14d8516a5e [term 1 FOLLOWER]: Leader election won for term 1
I20260925 22:48:45.562463 31239 raft_consensus.cc:686] T 18f5931027914ecb954e474b9d46b1ac P 40d68243fa5344f78b05bd14d8516a5e [term 1 LEADER]: Becoming Leader. State: Replica: 40d68243fa5344f78b05bd14d8516a5e, State: Running, Role: LEADER
I20260925 22:48:45.562830 31239 consensus_queue.cc:237] T 18f5931027914ecb954e474b9d46b1ac P 40d68243fa5344f78b05bd14d8516a5e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40d68243fa5344f78b05bd14d8516a5e" member_type: VOTER last_known_addr { host: "127.10.198.1" port: 44215 } } peers { permanent_uuid: "42d4fd97491841348de7dae820f1013b" member_type: VOTER last_known_addr { host: "127.10.198.2" port: 41859 } } peers { permanent_uuid: "03650d60099948d9b1464bb000859dc9" member_type: VOTER last_known_addr { host: "127.10.198.3" port: 42937 } }
I20260925 22:48:45.566399 30763 catalog_manager.cc:5495] T 18f5931027914ecb954e474b9d46b1ac P 40d68243fa5344f78b05bd14d8516a5e reported cstate change: term changed from 0 to 1, leader changed from <none> to 40d68243fa5344f78b05bd14d8516a5e (127.10.198.1). New cstate: current_term: 1 leader_uuid: "40d68243fa5344f78b05bd14d8516a5e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40d68243fa5344f78b05bd14d8516a5e" member_type: VOTER last_known_addr { host: "127.10.198.1" port: 44215 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "42d4fd97491841348de7dae820f1013b" member_type: VOTER last_known_addr { host: "127.10.198.2" port: 41859 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "03650d60099948d9b1464bb000859dc9" member_type: VOTER last_known_addr { host: "127.10.198.3" port: 42937 } health_report { overall_health: UNKNOWN } } }
W20260925 22:48:45.573256 31084 tablet.cc:2369] T 18f5931027914ecb954e474b9d46b1ac P 42d4fd97491841348de7dae820f1013b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 22:48:45.588698 31226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:48:45.584448 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40806 (local address 127.10.198.1:44215)
0925 22:48:45.584607 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0925 22:48:45.584613 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 22:48:45.584630 (+ 17us) server_negotiation.cc:407] Connection header received
0925 22:48:45.584743 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:48:45.584746 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:48:45.584817 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:48:45.584928 (+ 111us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 22:48:45.585163 (+ 235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:48:45.585945 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:48:45.586869 (+ 924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:48:45.587229 (+ 360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:48:45.587334 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:48:45.587637 (+ 303us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 22:48:45.588267 (+ 630us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 22:48:45.588397 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0925 22:48:45.588484 (+ 87us) server_negotiation.cc:299] Negotiation successful
0925 22:48:45.588552 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260925 22:48:45.595373 31168 raft_consensus.cc:1261] T 18f5931027914ecb954e474b9d46b1ac P 03650d60099948d9b1464bb000859dc9 [term 1 FOLLOWER]: Refusing update from remote peer 40d68243fa5344f78b05bd14d8516a5e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 22:48:45.595373 31037 raft_consensus.cc:1261] T 18f5931027914ecb954e474b9d46b1ac P 42d4fd97491841348de7dae820f1013b [term 1 FOLLOWER]: Refusing update from remote peer 40d68243fa5344f78b05bd14d8516a5e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 22:48:45.596367 31242 consensus_queue.cc:1035] T 18f5931027914ecb954e474b9d46b1ac P 40d68243fa5344f78b05bd14d8516a5e [LEADER]: Connected to new peer: Peer: permanent_uuid: "03650d60099948d9b1464bb000859dc9" member_type: VOTER last_known_addr { host: "127.10.198.3" port: 42937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 22:48:45.596611 31239 consensus_queue.cc:1035] T 18f5931027914ecb954e474b9d46b1ac P 40d68243fa5344f78b05bd14d8516a5e [LEADER]: Connected to new peer: Peer: permanent_uuid: "42d4fd97491841348de7dae820f1013b" member_type: VOTER last_known_addr { host: "127.10.198.2" port: 41859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 22:48:45.607364 31247 mvcc.cc:205] Tried to move back new op lower bound from 7333382248829980672 to 7333382248708390912. Current Snapshot: MvccSnapshot[applied={T|T < 7333382248829980672}]
I20260925 22:48:45.608441 31246 mvcc.cc:205] Tried to move back new op lower bound from 7333382248829980672 to 7333382248708390912. Current Snapshot: MvccSnapshot[applied={T|T < 7333382248829980672}]
I20260925 22:48:45.616036 30763 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:49320:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 22:48:45.616221 30763 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:49320:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 22:48:45.619135 30763 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f5fd62008eaf471c908be628985ba2e9: Sending DeleteTablet for 3 replicas of tablet 18f5931027914ecb954e474b9d46b1ac
I20260925 22:48:45.620222 30886 tablet_service.cc:1514] Processing DeleteTablet for tablet 18f5931027914ecb954e474b9d46b1ac with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 22:48:45 UTC) from {username='slave'} at 127.0.0.1:40794
I20260925 22:48:45.620494 31148 tablet_service.cc:1514] Processing DeleteTablet for tablet 18f5931027914ecb954e474b9d46b1ac with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 22:48:45 UTC) from {username='slave'} at 127.0.0.1:41966
I20260925 22:48:45.620824 31017 tablet_service.cc:1514] Processing DeleteTablet for tablet 18f5931027914ecb954e474b9d46b1ac with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 22:48:45 UTC) from {username='slave'} at 127.0.0.1:41868
I20260925 22:48:45.621062 31254 tablet_replica.cc:330] T 18f5931027914ecb954e474b9d46b1ac P 40d68243fa5344f78b05bd14d8516a5e: stopping tablet replica
I20260925 22:48:45.621443 11032 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrWXm8X/build/debug/bin/kudu with pid 30824
I20260925 22:48:45.621513 31254 raft_consensus.cc:2229] T 18f5931027914ecb954e474b9d46b1ac P 40d68243fa5344f78b05bd14d8516a5e [term 1 LEADER]: Raft consensus shutting down.
I20260925 22:48:45.628237 31255 tablet_replica.cc:330] T 18f5931027914ecb954e474b9d46b1ac P 03650d60099948d9b1464bb000859dc9: stopping tablet replica
I20260925 22:48:45.628715 31255 raft_consensus.cc:2229] T 18f5931027914ecb954e474b9d46b1ac P 03650d60099948d9b1464bb000859dc9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 22:48:45.628747 31256 tablet_replica.cc:330] T 18f5931027914ecb954e474b9d46b1ac P 42d4fd97491841348de7dae820f1013b: stopping tablet replica
I20260925 22:48:45.629156 31255 raft_consensus.cc:2258] T 18f5931027914ecb954e474b9d46b1ac P 03650d60099948d9b1464bb000859dc9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 22:48:45.629161 31256 raft_consensus.cc:2229] T 18f5931027914ecb954e474b9d46b1ac P 42d4fd97491841348de7dae820f1013b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 22:48:45.629526 31256 raft_consensus.cc:2258] T 18f5931027914ecb954e474b9d46b1ac P 42d4fd97491841348de7dae820f1013b [term 1 FOLLOWER]: Raft consensus is shut down!
W20260925 22:48:45.629940 30748 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.10.198.1:44215 (error 108)
I20260925 22:48:45.630163 11032 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrWXm8X/build/debug/bin/kudu with pid 30955
I20260925 22:48:45.631105 31255 ts_tablet_manager.cc:1905] T 18f5931027914ecb954e474b9d46b1ac P 03650d60099948d9b1464bb000859dc9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 22:48:45.632902 31225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:48:45.632511 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.198.1:44215 (local address 127.0.0.1:40816)
0925 22:48:45.632712 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0925 22:48:45.632802 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.198.1:44215: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
I20260925 22:48:45.633864 31255 ts_tablet_manager.cc:1918] T 18f5931027914ecb954e474b9d46b1ac P 03650d60099948d9b1464bb000859dc9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 22:48:45.633957 31255 log.cc:1196] T 18f5931027914ecb954e474b9d46b1ac P 03650d60099948d9b1464bb000859dc9: Deleting WAL directory at /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790376259650642-11032-0/minicluster-data/ts-2/wal/wals/18f5931027914ecb954e474b9d46b1ac
I20260925 22:48:45.634268 31255 ts_tablet_manager.cc:1939] T 18f5931027914ecb954e474b9d46b1ac P 03650d60099948d9b1464bb000859dc9: Deleting consensus metadata
I20260925 22:48:45.635656 30749 catalog_manager.cc:4841] TS 03650d60099948d9b1464bb000859dc9 (127.10.198.3:42937): tablet 18f5931027914ecb954e474b9d46b1ac (table test-table [id=e746ba589d2c443a80decfcdfc4ff7e9]) successfully deleted
W20260925 22:48:45.635761 30748 catalog_manager.cc:4584] TS 40d68243fa5344f78b05bd14d8516a5e (127.10.198.1:44215): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 18f5931027914ecb954e474b9d46b1ac: Network error: Client connection negotiation failed: client connection to 127.10.198.1:44215: connect: Connection refused (error 111)
I20260925 22:48:45.637517 11032 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrWXm8X/build/debug/bin/kudu with pid 31086
I20260925 22:48:45.639829 31225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:48:45.637729 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.198.2:41859 (local address 127.0.0.1:41874)
0925 22:48:45.639661 (+ 1932us) negotiation.cc:107] Waiting for socket to connect
0925 22:48:45.639734 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.198.2:41859: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1797}
I20260925 22:48:45.644403 11032 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrWXm8X/build/debug/bin/kudu with pid 30733
2026-09-25T22:48:45Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (1288 ms)
[----------] 6 tests from EncryptionPolicyTest (11009 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790376259650642-11032-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 22:48:45 dist-test-slave-5vdr krb5kdc[31260](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 22:48:45 dist-test-slave-5vdr krb5kdc[31260](info): set up 2 sockets
Sep 25 22:48:45 dist-test-slave-5vdr krb5kdc[31260](info): commencing operation
krb5kdc: starting...
W20260925 22:48:46.692201 11032 mini_kdc.cc:121] Time spent starting KDC: real 1.026s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790376259650642-11032-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790376259650642-11032-0/krb5kdc/test-user.keytab.
Sep 25 22:48:46 dist-test-slave-5vdr krb5kdc[31260](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376526, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T22:48:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T22:48:46Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790376259650642-11032-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790376259650642-11032-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 22:48:46.819504 11032 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrWXm8X/build/debug/bin/kudu
/tmp/dist-test-taskrWXm8X/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790376259650642-11032-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790376259650642-11032-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790376259650642-11032-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790376259650642-11032-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34097
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790376259650642-11032-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:41323
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34097
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790376259650642-11032-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790376259650642-11032-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790376259650642-11032-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:48:46.937397 31273 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:48:46.937723 31273 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:48:46.937769 31273 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:48:46.942133 31273 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 22:48:46.942220 31273 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:48:46.942241 31273 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:48:46.942258 31273 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 22:48:46.942273 31273 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 22:48:46.948082 31273 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41323
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790376259650642-11032-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790376259650642-11032-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34097
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790376259650642-11032-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34097
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790376259650642-11032-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31273
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790376259650642-11032-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:09:16 UTC on ccycloud.quasar-afmhyw.root.comops.site
build id 4673379
I20260925 22:48:46.949227 31273 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:48:46.950256 31273 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:48:46.959515 31281 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:48:46.959518 31279 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:48:46.959518 31278 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:48:46.959975 31273 server_base.cc:1034] running on GCE node
I20260925 22:48:46.960630 31273 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:48:46.961872 31273 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:48:46.963136 31273 hybrid_clock.cc:648] HybridClock initialized: now 1790376526963125 us; error 41 us; skew 500 ppm
Sep 25 22:48:46 dist-test-slave-5vdr krb5kdc[31260](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376526, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:48:46.966804 31273 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 22:48:46.967993 31273 webserver.cc:460] Webserver started at http://127.0.0.1:40291/ using document root <none> and password file <none>
I20260925 22:48:46.968636 31273 fs_manager.cc:359] Metadata directory not provided
I20260925 22:48:46.968740 31273 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:48:46.968987 31273 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:48:46.971295 31273 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790376259650642-11032-0/minicluster-data/master-0/data/instance:
uuid: "1acc8479832043d2bdbe013ac59d0b80"
format_stamp: "Formatted at 2026-09-25 22:48:46 on dist-test-slave-5vdr"
server_key: "3c86a26c44c98d496f77e2f4441eee7a"
server_key_iv: "2d8b877f2df682290c8e64b4e9278e46"
server_key_version: "encryptionkey@0"
I20260925 22:48:46.972035 31273 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790376259650642-11032-0/minicluster-data/master-0/wal/instance:
uuid: "1acc8479832043d2bdbe013ac59d0b80"
format_stamp: "Formatted at 2026-09-25 22:48:46 on dist-test-slave-5vdr"
server_key: "3c86a26c44c98d496f77e2f4441eee7a"
server_key_iv: "2d8b877f2df682290c8e64b4e9278e46"
server_key_version: "encryptionkey@0"
I20260925 22:48:46.976492 31273 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260925 22:48:46.979533 31288 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:48:46.981153 31273 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 22:48:46.981357 31273 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790376259650642-11032-0/minicluster-data/master-0/data,/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790376259650642-11032-0/minicluster-data/master-0/wal
uuid: "1acc8479832043d2bdbe013ac59d0b80"
format_stamp: "Formatted at 2026-09-25 22:48:46 on dist-test-slave-5vdr"
server_key: "3c86a26c44c98d496f77e2f4441eee7a"
server_key_iv: "2d8b877f2df682290c8e64b4e9278e46"
server_key_version: "encryptionkey@0"
I20260925 22:48:46.981518 31273 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790376259650642-11032-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790376259650642-11032-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790376259650642-11032-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:48:47.019071 31273 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:48:47.022887 31273 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:48:47.023160 31273 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:48:47.031358 31273 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34097
I20260925 22:48:47.031366 31340 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34097 every 8 connection(s)
I20260925 22:48:47.032588 31273 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790376259650642-11032-0/minicluster-data/master-0/data/info.pb
I20260925 22:48:47.035928 31341 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:48:47.037303 11032 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrWXm8X/build/debug/bin/kudu as pid 31273
I20260925 22:48:47.037431 11032 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790376259650642-11032-0/minicluster-data/master-0/wal/instance
I20260925 22:48:47.037752 11032 external_mini_cluster.cc:1428] Setting key 16ac88466ee3a763455dc8de6e34c450
I20260925 22:48:47.043625 31341 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1acc8479832043d2bdbe013ac59d0b80: Bootstrap starting.
I20260925 22:48:47.046736 31341 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1acc8479832043d2bdbe013ac59d0b80: Neither blocks nor log segments found. Creating new log.
I20260925 22:48:47.047946 31341 log.cc:824] T 00000000000000000000000000000000 P 1acc8479832043d2bdbe013ac59d0b80: Log is configured to *not* fsync() on all Append() calls
Sep 25 22:48:47 dist-test-slave-5vdr krb5kdc[31260](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376526, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260925 22:48:47.050491 31341 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1acc8479832043d2bdbe013ac59d0b80: No bootstrap required, opened a new log
I20260925 22:48:47.054589 31341 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1acc8479832043d2bdbe013ac59d0b80 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1acc8479832043d2bdbe013ac59d0b80" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34097 } }
I20260925 22:48:47.054898 31341 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1acc8479832043d2bdbe013ac59d0b80 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:48:47.054982 31341 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1acc8479832043d2bdbe013ac59d0b80 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1acc8479832043d2bdbe013ac59d0b80, State: Initialized, Role: FOLLOWER
I20260925 22:48:47.055465 31344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:48:47.039412 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53663 (local address 127.0.0.1:34097)
0925 22:48:47.039984 (+ 572us) server_negotiation.cc:206] Beginning negotiation
0925 22:48:47.040008 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0925 22:48:47.040058 (+ 50us) server_negotiation.cc:407] Connection header received
0925 22:48:47.040954 (+ 896us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:48:47.040980 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:48:47.041484 (+ 504us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:48:47.041912 (+ 428us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:48:47.042697 (+ 785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:48:47.044233 (+ 1536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:48:47.045276 (+ 1043us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:48:47.046025 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:48:47.046212 (+ 187us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:48:47.049722 (+ 3510us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:48:47.049728 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:48:47.049745 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:48:47.049778 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:48:47.051741 (+ 1963us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:48:47.052481 (+ 740us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:48:47.052488 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:48:47.052494 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:48:47.052630 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:48:47.053038 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:48:47.053049 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:48:47.053051 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:48:47.053523 (+ 472us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:48:47.053809 (+ 286us) server_negotiation.cc:1035] Waiting for connection context
0925 22:48:47.054193 (+ 384us) server_negotiation.cc:299] Negotiation successful
0925 22:48:47.054454 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":378,"thread_start_us":191,"threads_started":1}
I20260925 22:48:47.055466 31341 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1acc8479832043d2bdbe013ac59d0b80 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1acc8479832043d2bdbe013ac59d0b80" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34097 } }
I20260925 22:48:47.055882 31341 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1acc8479832043d2bdbe013ac59d0b80 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 22:48:47.055972 31341 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1acc8479832043d2bdbe013ac59d0b80 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 22:48:47.056092 31341 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1acc8479832043d2bdbe013ac59d0b80 [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:48:47.057363 31341 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1acc8479832043d2bdbe013ac59d0b80 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1acc8479832043d2bdbe013ac59d0b80" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34097 } }
I20260925 22:48:47.057796 31341 leader_election.cc:302] T 00000000000000000000000000000000 P 1acc8479832043d2bdbe013ac59d0b80 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1acc8479832043d2bdbe013ac59d0b80; no voters:
I20260925 22:48:47.058248 31341 leader_election.cc:288] T 00000000000000000000000000000000 P 1acc8479832043d2bdbe013ac59d0b80 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 22:48:47.058334 31346 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1acc8479832043d2bdbe013ac59d0b80 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 22:48:47.058802 31346 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1acc8479832043d2bdbe013ac59d0b80 [term 1 LEADER]: Becoming Leader. State: Replica: 1acc8479832043d2bdbe013ac59d0b80, State: Running, Role: LEADER
I20260925 22:48:47.059266 11032 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 22:48:47.059267 31346 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1acc8479832043d2bdbe013ac59d0b80 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1acc8479832043d2bdbe013ac59d0b80" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34097 } }
I20260925 22:48:47.059741 31341 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1acc8479832043d2bdbe013ac59d0b80 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 22:48:47.061105 31347 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1acc8479832043d2bdbe013ac59d0b80 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1acc8479832043d2bdbe013ac59d0b80" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1acc8479832043d2bdbe013ac59d0b80" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34097 } } }
I20260925 22:48:47.061257 31347 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1acc8479832043d2bdbe013ac59d0b80 [sys.catalog]: This master's current role is: LEADER
I20260925 22:48:47.061513 31347 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1acc8479832043d2bdbe013ac59d0b80 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1acc8479832043d2bdbe013ac59d0b80. Latest consensus state: current_term: 1 leader_uuid: "1acc8479832043d2bdbe013ac59d0b80" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1acc8479832043d2bdbe013ac59d0b80" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34097 } } }
I20260925 22:48:47.061641 31347 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1acc8479832043d2bdbe013ac59d0b80 [sys.catalog]: This master's current role is: LEADER
W20260925 22:48:47.065085 31365 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 1acc8479832043d2bdbe013ac59d0b80: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 22:48:47.065191 31365 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 22:48:47.065258 31360 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 22:48:47.066475 31360 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 22:48:47.073601 31360 catalog_manager.cc:1347] Generated new cluster ID: f45dc90f05d2494dbc94cb852c8897d2
I20260925 22:48:47.073704 31360 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 22:48:47.073755 31344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:48:47.061653 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35043 (local address 127.0.0.1:34097)
0925 22:48:47.061786 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0925 22:48:47.061791 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:48:47.063494 (+ 1703us) server_negotiation.cc:407] Connection header received
0925 22:48:47.063633 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:48:47.063637 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:48:47.063692 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:48:47.063808 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:48:47.064772 (+ 964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:48:47.065524 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:48:47.066804 (+ 1280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:48:47.067144 (+ 340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:48:47.067268 (+ 124us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:48:47.069200 (+ 1932us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:48:47.069204 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:48:47.069208 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:48:47.069242 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:48:47.071097 (+ 1855us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:48:47.071797 (+ 700us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:48:47.071801 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:48:47.071803 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:48:47.071958 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:48:47.072387 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:48:47.072391 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:48:47.072393 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:48:47.072665 (+ 272us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:48:47.072825 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0925 22:48:47.073373 (+ 548us) server_negotiation.cc:299] Negotiation successful
0925 22:48:47.073501 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260925 22:48:47.095495 31360 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 22:48:47.096649 31360 catalog_manager.cc:1504] Loading token signing keys...
I20260925 22:48:47.110972 31360 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1acc8479832043d2bdbe013ac59d0b80: Generated new TSK 0
I20260925 22:48:47.111877 31360 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 22:48:47.120024 11032 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrWXm8X/build/debug/bin/kudu with pid 31273
W20260925 22:48:47.125934 31258 connection.cc:535] client connection to 127.0.0.1:34097 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:34097): Connection reset by peer (error 104)
2026-09-25T22:48:47Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (1473 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790376259650642-11032-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 22:48:47 dist-test-slave-5vdr krb5kdc[31372](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 22:48:47 dist-test-slave-5vdr krb5kdc[31372](info): set up 2 sockets
Sep 25 22:48:47 dist-test-slave-5vdr krb5kdc[31372](info): commencing operation
krb5kdc: starting...
W20260925 22:48:48.172197 11032 mini_kdc.cc:121] Time spent starting KDC: real 1.031s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790376259650642-11032-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790376259650642-11032-0/krb5kdc/test-user.keytab.
Sep 25 22:48:48 dist-test-slave-5vdr krb5kdc[31372](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376528, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T22:48:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T22:48:48Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790376259650642-11032-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790376259650642-11032-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 22:48:48.308302 11032 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrWXm8X/build/debug/bin/kudu
/tmp/dist-test-taskrWXm8X/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790376259650642-11032-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790376259650642-11032-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790376259650642-11032-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790376259650642-11032-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38305
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790376259650642-11032-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:33247
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38305
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790376259650642-11032-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790376259650642-11032-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790376259650642-11032-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:48:48.425767 31385 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:48:48.426079 31385 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:48:48.426123 31385 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:48:48.430680 31385 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 22:48:48.430753 31385 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:48:48.430774 31385 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:48:48.430794 31385 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 22:48:48.430810 31385 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 22:48:48.436884 31385 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33247
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790376259650642-11032-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790376259650642-11032-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38305
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790376259650642-11032-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38305
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790376259650642-11032-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31385
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790376259650642-11032-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:09:16 UTC on ccycloud.quasar-afmhyw.root.comops.site
build id 4673379
I20260925 22:48:48.438064 31385 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:48:48.439152 31385 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:48:48.449242 31393 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:48:48.449242 31391 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:48:48.449208 31390 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:48:48.449319 31385 server_base.cc:1034] running on GCE node
I20260925 22:48:48.450176 31385 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:48:48.451479 31385 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:48:48.452705 31385 hybrid_clock.cc:648] HybridClock initialized: now 1790376528452683 us; error 44 us; skew 500 ppm
Sep 25 22:48:48 dist-test-slave-5vdr krb5kdc[31372](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376528, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:48:48.456269 31385 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 22:48:48.457458 31385 webserver.cc:460] Webserver started at http://127.0.0.1:44503/ using document root <none> and password file <none>
I20260925 22:48:48.458148 31385 fs_manager.cc:359] Metadata directory not provided
I20260925 22:48:48.458230 31385 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:48:48.458516 31385 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:48:48.460975 31385 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790376259650642-11032-0/minicluster-data/master-0/data/instance:
uuid: "36f608a50f6845dcafc750b58c3fb767"
format_stamp: "Formatted at 2026-09-25 22:48:48 on dist-test-slave-5vdr"
server_key: "5f2f9d38b2a2f865e4f0fc12da7d54cf"
server_key_iv: "51c6d845a425b45e20e7b001df39b877"
server_key_version: "encryptionkey@0"
I20260925 22:48:48.461694 31385 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790376259650642-11032-0/minicluster-data/master-0/wal/instance:
uuid: "36f608a50f6845dcafc750b58c3fb767"
format_stamp: "Formatted at 2026-09-25 22:48:48 on dist-test-slave-5vdr"
server_key: "5f2f9d38b2a2f865e4f0fc12da7d54cf"
server_key_iv: "51c6d845a425b45e20e7b001df39b877"
server_key_version: "encryptionkey@0"
I20260925 22:48:48.466374 31385 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260925 22:48:48.469600 31400 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:48:48.471076 31385 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260925 22:48:48.471259 31385 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790376259650642-11032-0/minicluster-data/master-0/data,/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790376259650642-11032-0/minicluster-data/master-0/wal
uuid: "36f608a50f6845dcafc750b58c3fb767"
format_stamp: "Formatted at 2026-09-25 22:48:48 on dist-test-slave-5vdr"
server_key: "5f2f9d38b2a2f865e4f0fc12da7d54cf"
server_key_iv: "51c6d845a425b45e20e7b001df39b877"
server_key_version: "encryptionkey@0"
I20260925 22:48:48.471400 31385 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790376259650642-11032-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790376259650642-11032-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790376259650642-11032-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:48:48.497191 31385 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:48:48.500727 31385 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:48:48.500908 31385 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:48:48.508526 31385 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38305
I20260925 22:48:48.508539 31452 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38305 every 8 connection(s)
I20260925 22:48:48.509603 31385 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790376259650642-11032-0/minicluster-data/master-0/data/info.pb
I20260925 22:48:48.512750 31453 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:48:48.516028 11032 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrWXm8X/build/debug/bin/kudu as pid 31385
I20260925 22:48:48.516202 11032 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790376259650642-11032-0/minicluster-data/master-0/wal/instance
I20260925 22:48:48.516510 11032 external_mini_cluster.cc:1428] Setting key 7505b7129888d24fcedad638f0577ee5
I20260925 22:48:48.519339 31453 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 36f608a50f6845dcafc750b58c3fb767: Bootstrap starting.
I20260925 22:48:48.522158 31453 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 36f608a50f6845dcafc750b58c3fb767: Neither blocks nor log segments found. Creating new log.
I20260925 22:48:48.523123 31453 log.cc:824] T 00000000000000000000000000000000 P 36f608a50f6845dcafc750b58c3fb767: Log is configured to *not* fsync() on all Append() calls
I20260925 22:48:48.525854 31453 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 36f608a50f6845dcafc750b58c3fb767: No bootstrap required, opened a new log
I20260925 22:48:48.526877 31456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:48:48.518010 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35735 (local address 127.0.0.1:38305)
0925 22:48:48.518494 (+ 484us) server_negotiation.cc:206] Beginning negotiation
0925 22:48:48.518514 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0925 22:48:48.518553 (+ 39us) server_negotiation.cc:407] Connection header received
0925 22:48:48.519221 (+ 668us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:48:48.519240 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:48:48.519684 (+ 444us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:48:48.520091 (+ 407us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:48:48.520446 (+ 355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:48:48.521479 (+ 1033us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:48:48.522798 (+ 1319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:48:48.523430 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:48:48.523605 (+ 175us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:48:48.524637 (+ 1032us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:48:48.524644 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:48:48.524663 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 22:48:48.525235 (+ 572us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:48:48.525317 (+ 82us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 22:48:48.525349 (+ 32us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:48:48.525510 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0925 22:48:48.525678 (+ 168us) server_negotiation.cc:299] Negotiation successful
0925 22:48:48.525828 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":320,"thread_start_us":167,"threads_started":1}
I20260925 22:48:48.530151 11032 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 22:48:48.530032 31453 raft_consensus.cc:348] T 00000000000000000000000000000000 P 36f608a50f6845dcafc750b58c3fb767 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36f608a50f6845dcafc750b58c3fb767" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38305 } }
I20260925 22:48:48.530372 31453 raft_consensus.cc:374] T 00000000000000000000000000000000 P 36f608a50f6845dcafc750b58c3fb767 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:48:48.530427 31453 raft_consensus.cc:729] T 00000000000000000000000000000000 P 36f608a50f6845dcafc750b58c3fb767 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 36f608a50f6845dcafc750b58c3fb767, State: Initialized, Role: FOLLOWER
I20260925 22:48:48.530946 31453 consensus_queue.cc:260] T 00000000000000000000000000000000 P 36f608a50f6845dcafc750b58c3fb767 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36f608a50f6845dcafc750b58c3fb767" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38305 } }
I20260925 22:48:48.531142 31453 raft_consensus.cc:388] T 00000000000000000000000000000000 P 36f608a50f6845dcafc750b58c3fb767 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 22:48:48.531216 31453 raft_consensus.cc:482] T 00000000000000000000000000000000 P 36f608a50f6845dcafc750b58c3fb767 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 22:48:48.531320 31453 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 36f608a50f6845dcafc750b58c3fb767 [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:48:48.532709 31453 raft_consensus.cc:504] T 00000000000000000000000000000000 P 36f608a50f6845dcafc750b58c3fb767 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36f608a50f6845dcafc750b58c3fb767" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38305 } }
I20260925 22:48:48.533169 31453 leader_election.cc:302] T 00000000000000000000000000000000 P 36f608a50f6845dcafc750b58c3fb767 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 36f608a50f6845dcafc750b58c3fb767; no voters:
I20260925 22:48:48.533583 31453 leader_election.cc:288] T 00000000000000000000000000000000 P 36f608a50f6845dcafc750b58c3fb767 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 22:48:48.533696 31463 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 36f608a50f6845dcafc750b58c3fb767 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 22:48:48.533931 31463 raft_consensus.cc:686] T 00000000000000000000000000000000 P 36f608a50f6845dcafc750b58c3fb767 [term 1 LEADER]: Becoming Leader. State: Replica: 36f608a50f6845dcafc750b58c3fb767, State: Running, Role: LEADER
I20260925 22:48:48.534279 31463 consensus_queue.cc:237] T 00000000000000000000000000000000 P 36f608a50f6845dcafc750b58c3fb767 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36f608a50f6845dcafc750b58c3fb767" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38305 } }
I20260925 22:48:48.535120 31453 sys_catalog.cc:564] T 00000000000000000000000000000000 P 36f608a50f6845dcafc750b58c3fb767 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 22:48:48.536069 31464 sys_catalog.cc:455] T 00000000000000000000000000000000 P 36f608a50f6845dcafc750b58c3fb767 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "36f608a50f6845dcafc750b58c3fb767" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36f608a50f6845dcafc750b58c3fb767" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38305 } } }
I20260925 22:48:48.536259 31464 sys_catalog.cc:458] T 00000000000000000000000000000000 P 36f608a50f6845dcafc750b58c3fb767 [sys.catalog]: This master's current role is: LEADER
I20260925 22:48:48.536588 31465 sys_catalog.cc:455] T 00000000000000000000000000000000 P 36f608a50f6845dcafc750b58c3fb767 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 36f608a50f6845dcafc750b58c3fb767. Latest consensus state: current_term: 1 leader_uuid: "36f608a50f6845dcafc750b58c3fb767" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36f608a50f6845dcafc750b58c3fb767" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38305 } } }
I20260925 22:48:48.536707 31465 sys_catalog.cc:458] T 00000000000000000000000000000000 P 36f608a50f6845dcafc750b58c3fb767 [sys.catalog]: This master's current role is: LEADER
I20260925 22:48:48.537964 31456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:48:48.532679 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48553 (local address 127.0.0.1:38305)
0925 22:48:48.532830 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0925 22:48:48.532837 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 22:48:48.533007 (+ 170us) server_negotiation.cc:407] Connection header received
0925 22:48:48.533133 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:48:48.533135 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:48:48.533180 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:48:48.533249 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:48:48.533631 (+ 382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:48:48.534425 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:48:48.535611 (+ 1186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:48:48.536022 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:48:48.536149 (+ 127us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:48:48.537376 (+ 1227us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:48:48.537382 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:48:48.537388 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 22:48:48.537404 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:48:48.537440 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 22:48:48.537457 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:48:48.537564 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0925 22:48:48.537788 (+ 224us) server_negotiation.cc:299] Negotiation successful
0925 22:48:48.537833 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
W20260925 22:48:48.540898 31478 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 36f608a50f6845dcafc750b58c3fb767: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 22:48:48.541014 31478 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 22:48:48.541131 31472 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 22:48:48.542130 31472 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 22:48:48.548774 31472 catalog_manager.cc:1347] Generated new cluster ID: 5cf30820004a401f80d4c9e53f2fa693
I20260925 22:48:48.548898 31472 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 22:48:48.559670 31472 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 22:48:48.560797 31472 catalog_manager.cc:1504] Loading token signing keys...
I20260925 22:48:48.569732 31472 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 36f608a50f6845dcafc750b58c3fb767: Generated new TSK 0
I20260925 22:48:48.570568 31472 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 22:48:48.574991 11032 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrWXm8X/build/debug/bin/kudu with pid 31385
W20260925 22:48:48.580816 31370 connection.cc:535] client connection to 127.0.0.1:38305 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:38305): Connection reset by peer (error 104)
2026-09-25T22:48:48Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1455 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790376259650642-11032-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 22:48:48 dist-test-slave-5vdr krb5kdc[31484](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 22:48:48 dist-test-slave-5vdr krb5kdc[31484](info): set up 2 sockets
Sep 25 22:48:48 dist-test-slave-5vdr krb5kdc[31484](info): commencing operation
krb5kdc: starting...
W20260925 22:48:49.632254 11032 mini_kdc.cc:121] Time spent starting KDC: real 1.036s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790376259650642-11032-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790376259650642-11032-0/krb5kdc/test-user.keytab.
Sep 25 22:48:49 dist-test-slave-5vdr krb5kdc[31484](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376529, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T22:48:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T22:48:49Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790376259650642-11032-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790376259650642-11032-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 22:48:49.761250 11032 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrWXm8X/build/debug/bin/kudu
/tmp/dist-test-taskrWXm8X/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790376259650642-11032-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790376259650642-11032-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790376259650642-11032-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790376259650642-11032-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39391
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790376259650642-11032-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35195
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39391
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790376259650642-11032-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790376259650642-11032-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790376259650642-11032-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:48:49.883033 31497 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:48:49.883399 31497 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:48:49.883519 31497 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:48:49.887964 31497 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 22:48:49.888068 31497 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:48:49.888111 31497 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:48:49.888167 31497 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 22:48:49.888221 31497 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 22:48:49.894161 31497 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35195
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790376259650642-11032-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790376259650642-11032-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39391
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790376259650642-11032-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39391
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790376259650642-11032-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31497
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790376259650642-11032-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:09:16 UTC on ccycloud.quasar-afmhyw.root.comops.site
build id 4673379
I20260925 22:48:49.895496 31497 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:48:49.896708 31497 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:48:49.906080 31503 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:48:49.906179 31505 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:48:49.906064 31502 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:48:49.906692 31497 server_base.cc:1034] running on GCE node
I20260925 22:48:49.907370 31497 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:48:49.908844 31497 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:48:49.910104 31497 hybrid_clock.cc:648] HybridClock initialized: now 1790376529910087 us; error 35 us; skew 500 ppm
Sep 25 22:48:49 dist-test-slave-5vdr krb5kdc[31484](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376529, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:48:49.918789 31497 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 22:48:49.920094 31497 webserver.cc:460] Webserver started at http://127.0.0.1:41651/ using document root <none> and password file <none>
I20260925 22:48:49.920751 31497 fs_manager.cc:359] Metadata directory not provided
I20260925 22:48:49.920861 31497 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:48:49.921126 31497 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:48:49.923441 31497 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790376259650642-11032-0/minicluster-data/master-0/data/instance:
uuid: "9ee9a100a3c54cfd87d47e725a831d18"
format_stamp: "Formatted at 2026-09-25 22:48:49 on dist-test-slave-5vdr"
server_key: "5a73d40879823c3e137f04079bf1e918"
server_key_iv: "40091f5f41b4b20337b8866a02190ce6"
server_key_version: "encryptionkey@0"
I20260925 22:48:49.924180 31497 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790376259650642-11032-0/minicluster-data/master-0/wal/instance:
uuid: "9ee9a100a3c54cfd87d47e725a831d18"
format_stamp: "Formatted at 2026-09-25 22:48:49 on dist-test-slave-5vdr"
server_key: "5a73d40879823c3e137f04079bf1e918"
server_key_iv: "40091f5f41b4b20337b8866a02190ce6"
server_key_version: "encryptionkey@0"
I20260925 22:48:49.928628 31497 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260925 22:48:49.931589 31512 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:48:49.933259 31497 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 22:48:49.933470 31497 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790376259650642-11032-0/minicluster-data/master-0/data,/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790376259650642-11032-0/minicluster-data/master-0/wal
uuid: "9ee9a100a3c54cfd87d47e725a831d18"
format_stamp: "Formatted at 2026-09-25 22:48:49 on dist-test-slave-5vdr"
server_key: "5a73d40879823c3e137f04079bf1e918"
server_key_iv: "40091f5f41b4b20337b8866a02190ce6"
server_key_version: "encryptionkey@0"
I20260925 22:48:49.933635 31497 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790376259650642-11032-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790376259650642-11032-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790376259650642-11032-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:48:49.939019 31497 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:48:49.942444 31497 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:48:49.942689 31497 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:48:49.951366 31497 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39391
I20260925 22:48:49.951375 31564 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39391 every 8 connection(s)
I20260925 22:48:49.952580 31497 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790376259650642-11032-0/minicluster-data/master-0/data/info.pb
I20260925 22:48:49.955687 31565 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:48:49.958840 11032 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrWXm8X/build/debug/bin/kudu as pid 31497
I20260925 22:48:49.958981 11032 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790376259650642-11032-0/minicluster-data/master-0/wal/instance
I20260925 22:48:49.959383 11032 external_mini_cluster.cc:1428] Setting key 7059fe2253a8161439552e2db1dbc332
I20260925 22:48:49.962738 31565 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9ee9a100a3c54cfd87d47e725a831d18: Bootstrap starting.
I20260925 22:48:49.965503 31568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:48:49.961089 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48539 (local address 127.0.0.1:39391)
0925 22:48:49.961603 (+ 514us) server_negotiation.cc:206] Beginning negotiation
0925 22:48:49.961619 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0925 22:48:49.961660 (+ 41us) server_negotiation.cc:407] Connection header received
0925 22:48:49.962350 (+ 690us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:48:49.962371 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:48:49.962712 (+ 341us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:48:49.963105 (+ 393us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:48:49.964115 (+ 1010us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:48:49.964120 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:48:49.964134 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 22:48:49.964146 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:48:49.964219 (+ 73us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 22:48:49.964241 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:48:49.964360 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0925 22:48:49.964479 (+ 119us) server_negotiation.cc:299] Negotiation successful
0925 22:48:49.964676 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":333,"thread_start_us":167,"threads_started":1}
I20260925 22:48:49.965849 31565 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9ee9a100a3c54cfd87d47e725a831d18: Neither blocks nor log segments found. Creating new log.
I20260925 22:48:49.966665 31565 log.cc:824] T 00000000000000000000000000000000 P 9ee9a100a3c54cfd87d47e725a831d18: Log is configured to *not* fsync() on all Append() calls
I20260925 22:48:49.968838 11032 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 22:48:49.969017 31565 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9ee9a100a3c54cfd87d47e725a831d18: No bootstrap required, opened a new log
I20260925 22:48:49.973114 31568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:48:49.971238 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43775 (local address 127.0.0.1:39391)
0925 22:48:49.971392 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0925 22:48:49.971398 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 22:48:49.971574 (+ 176us) server_negotiation.cc:407] Connection header received
0925 22:48:49.971713 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:48:49.971715 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:48:49.971751 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:48:49.971822 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:48:49.972618 (+ 796us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:48:49.972621 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:48:49.972623 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 22:48:49.972634 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:48:49.972666 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 22:48:49.972678 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:48:49.972771 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0925 22:48:49.972905 (+ 134us) server_negotiation.cc:299] Negotiation successful
0925 22:48:49.972941 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260925 22:48:49.974191 31565 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9ee9a100a3c54cfd87d47e725a831d18 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ee9a100a3c54cfd87d47e725a831d18" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39391 } }
I20260925 22:48:49.974478 31565 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9ee9a100a3c54cfd87d47e725a831d18 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:48:49.974565 31565 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9ee9a100a3c54cfd87d47e725a831d18 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9ee9a100a3c54cfd87d47e725a831d18, State: Initialized, Role: FOLLOWER
I20260925 22:48:49.975008 31565 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9ee9a100a3c54cfd87d47e725a831d18 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ee9a100a3c54cfd87d47e725a831d18" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39391 } }
I20260925 22:48:49.975168 31565 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9ee9a100a3c54cfd87d47e725a831d18 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 22:48:49.975238 31565 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9ee9a100a3c54cfd87d47e725a831d18 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 22:48:49.975370 31565 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9ee9a100a3c54cfd87d47e725a831d18 [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:48:49.976528 31565 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9ee9a100a3c54cfd87d47e725a831d18 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ee9a100a3c54cfd87d47e725a831d18" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39391 } }
I20260925 22:48:49.976969 31565 leader_election.cc:302] T 00000000000000000000000000000000 P 9ee9a100a3c54cfd87d47e725a831d18 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9ee9a100a3c54cfd87d47e725a831d18; no voters:
I20260925 22:48:49.977377 31565 leader_election.cc:288] T 00000000000000000000000000000000 P 9ee9a100a3c54cfd87d47e725a831d18 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 22:48:49.977458 31575 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9ee9a100a3c54cfd87d47e725a831d18 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 22:48:49.977732 31575 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9ee9a100a3c54cfd87d47e725a831d18 [term 1 LEADER]: Becoming Leader. State: Replica: 9ee9a100a3c54cfd87d47e725a831d18, State: Running, Role: LEADER
I20260925 22:48:49.978096 31575 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9ee9a100a3c54cfd87d47e725a831d18 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ee9a100a3c54cfd87d47e725a831d18" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39391 } }
I20260925 22:48:49.978930 31565 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9ee9a100a3c54cfd87d47e725a831d18 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 22:48:49.979866 31577 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9ee9a100a3c54cfd87d47e725a831d18 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9ee9a100a3c54cfd87d47e725a831d18. Latest consensus state: current_term: 1 leader_uuid: "9ee9a100a3c54cfd87d47e725a831d18" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ee9a100a3c54cfd87d47e725a831d18" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39391 } } }
I20260925 22:48:49.980034 31577 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9ee9a100a3c54cfd87d47e725a831d18 [sys.catalog]: This master's current role is: LEADER
I20260925 22:48:49.980402 31584 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 22:48:49.980834 31576 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9ee9a100a3c54cfd87d47e725a831d18 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9ee9a100a3c54cfd87d47e725a831d18" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ee9a100a3c54cfd87d47e725a831d18" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39391 } } }
I20260925 22:48:49.981024 31576 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9ee9a100a3c54cfd87d47e725a831d18 [sys.catalog]: This master's current role is: LEADER
I20260925 22:48:49.983819 31584 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 22:48:49.990175 31584 catalog_manager.cc:1347] Generated new cluster ID: 43d6827af8a94d938e198d10f7f0201f
I20260925 22:48:49.990285 31584 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 22:48:50.004412 31584 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 22:48:50.005015 31584 catalog_manager.cc:1504] Loading token signing keys...
I20260925 22:48:50.014772 31584 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9ee9a100a3c54cfd87d47e725a831d18: Generated new TSK 0
I20260925 22:48:50.015640 31584 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 22:48:50.025111 11032 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrWXm8X/build/debug/bin/kudu with pid 31497
2026-09-25T22:48:50Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1449 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790376259650642-11032-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 22:48:50 dist-test-slave-5vdr krb5kdc[31596](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 22:48:50 dist-test-slave-5vdr krb5kdc[31596](info): set up 2 sockets
Sep 25 22:48:50 dist-test-slave-5vdr krb5kdc[31596](info): commencing operation
krb5kdc: starting...
W20260925 22:48:51.076300 11032 mini_kdc.cc:121] Time spent starting KDC: real 1.031s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790376259650642-11032-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790376259650642-11032-0/krb5kdc/test-user.keytab.
Sep 25 22:48:51 dist-test-slave-5vdr krb5kdc[31596](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376531, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T22:48:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T22:48:51Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790376259650642-11032-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790376259650642-11032-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 22:48:51.211501 11032 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrWXm8X/build/debug/bin/kudu
/tmp/dist-test-taskrWXm8X/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790376259650642-11032-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790376259650642-11032-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790376259650642-11032-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790376259650642-11032-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45623
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790376259650642-11032-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36253
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45623
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790376259650642-11032-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790376259650642-11032-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790376259650642-11032-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:48:51.334189 31609 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:48:51.334514 31609 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:48:51.334559 31609 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:48:51.338881 31609 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 22:48:51.338969 31609 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:48:51.338990 31609 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:48:51.339008 31609 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 22:48:51.339025 31609 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 22:48:51.345000 31609 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36253
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790376259650642-11032-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790376259650642-11032-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45623
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790376259650642-11032-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45623
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790376259650642-11032-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31609
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790376259650642-11032-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:09:16 UTC on ccycloud.quasar-afmhyw.root.comops.site
build id 4673379
I20260925 22:48:51.346216 31609 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:48:51.347251 31609 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:48:51.356403 31615 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:48:51.356403 31614 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:48:51.356612 31609 server_base.cc:1034] running on GCE node
W20260925 22:48:51.356407 31617 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:48:51.357321 31609 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:48:51.358669 31609 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:48:51.359889 31609 hybrid_clock.cc:648] HybridClock initialized: now 1790376531359826 us; error 39 us; skew 500 ppm
Sep 25 22:48:51 dist-test-slave-5vdr krb5kdc[31596](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376531, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:48:51.362910 31609 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 22:48:51.364066 31609 webserver.cc:460] Webserver started at http://127.0.0.1:46485/ using document root <none> and password file <none>
I20260925 22:48:51.364717 31609 fs_manager.cc:359] Metadata directory not provided
I20260925 22:48:51.364827 31609 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:48:51.365087 31609 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:48:51.367364 31609 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790376259650642-11032-0/minicluster-data/master-0/data/instance:
uuid: "8f498fb67e5c4bb985359f0faf71c320"
format_stamp: "Formatted at 2026-09-25 22:48:51 on dist-test-slave-5vdr"
server_key: "3dd63085bcd13ab6716a4dbc9458e4b0"
server_key_iv: "492c20b66b7670d65a02c7cbe0de2ef3"
server_key_version: "encryptionkey@0"
I20260925 22:48:51.368091 31609 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790376259650642-11032-0/minicluster-data/master-0/wal/instance:
uuid: "8f498fb67e5c4bb985359f0faf71c320"
format_stamp: "Formatted at 2026-09-25 22:48:51 on dist-test-slave-5vdr"
server_key: "3dd63085bcd13ab6716a4dbc9458e4b0"
server_key_iv: "492c20b66b7670d65a02c7cbe0de2ef3"
server_key_version: "encryptionkey@0"
I20260925 22:48:51.372465 31609 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.003s
I20260925 22:48:51.375347 31624 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:48:51.376855 31609 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 22:48:51.377061 31609 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790376259650642-11032-0/minicluster-data/master-0/data,/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790376259650642-11032-0/minicluster-data/master-0/wal
uuid: "8f498fb67e5c4bb985359f0faf71c320"
format_stamp: "Formatted at 2026-09-25 22:48:51 on dist-test-slave-5vdr"
server_key: "3dd63085bcd13ab6716a4dbc9458e4b0"
server_key_iv: "492c20b66b7670d65a02c7cbe0de2ef3"
server_key_version: "encryptionkey@0"
I20260925 22:48:51.377209 31609 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790376259650642-11032-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790376259650642-11032-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790376259650642-11032-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:48:51.389070 31609 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:48:51.392777 31609 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:48:51.393026 31609 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:48:51.400493 31609 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45623
I20260925 22:48:51.400516 31676 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45623 every 8 connection(s)
I20260925 22:48:51.401567 31609 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790376259650642-11032-0/minicluster-data/master-0/data/info.pb
I20260925 22:48:51.405040 31677 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:48:51.409632 11032 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrWXm8X/build/debug/bin/kudu as pid 31609
I20260925 22:48:51.409786 11032 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790376259650642-11032-0/minicluster-data/master-0/wal/instance
I20260925 22:48:51.410104 11032 external_mini_cluster.cc:1428] Setting key 17fc1aaf96fb109c5b406796be72ce9a
I20260925 22:48:51.412015 31677 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8f498fb67e5c4bb985359f0faf71c320: Bootstrap starting.
I20260925 22:48:51.415117 31677 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8f498fb67e5c4bb985359f0faf71c320: Neither blocks nor log segments found. Creating new log.
I20260925 22:48:51.416347 31677 log.cc:824] T 00000000000000000000000000000000 P 8f498fb67e5c4bb985359f0faf71c320: Log is configured to *not* fsync() on all Append() calls
I20260925 22:48:51.419986 31677 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8f498fb67e5c4bb985359f0faf71c320: No bootstrap required, opened a new log
Sep 25 22:48:51 dist-test-slave-5vdr krb5kdc[31596](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376531, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260925 22:48:51.423599 31677 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8f498fb67e5c4bb985359f0faf71c320 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f498fb67e5c4bb985359f0faf71c320" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45623 } }
I20260925 22:48:51.424057 31677 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8f498fb67e5c4bb985359f0faf71c320 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:48:51.424155 31677 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8f498fb67e5c4bb985359f0faf71c320 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f498fb67e5c4bb985359f0faf71c320, State: Initialized, Role: FOLLOWER
I20260925 22:48:51.424762 31677 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8f498fb67e5c4bb985359f0faf71c320 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f498fb67e5c4bb985359f0faf71c320" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45623 } }
I20260925 22:48:51.424995 31677 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8f498fb67e5c4bb985359f0faf71c320 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 22:48:51.425082 31677 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8f498fb67e5c4bb985359f0faf71c320 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 22:48:51.425230 31677 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8f498fb67e5c4bb985359f0faf71c320 [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:48:51.426296 31677 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8f498fb67e5c4bb985359f0faf71c320 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f498fb67e5c4bb985359f0faf71c320" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45623 } }
I20260925 22:48:51.426715 31677 leader_election.cc:302] T 00000000000000000000000000000000 P 8f498fb67e5c4bb985359f0faf71c320 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8f498fb67e5c4bb985359f0faf71c320; no voters:
I20260925 22:48:51.427098 31677 leader_election.cc:288] T 00000000000000000000000000000000 P 8f498fb67e5c4bb985359f0faf71c320 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 22:48:51.427512 31682 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8f498fb67e5c4bb985359f0faf71c320 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 22:48:51.427738 31682 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8f498fb67e5c4bb985359f0faf71c320 [term 1 LEADER]: Becoming Leader. State: Replica: 8f498fb67e5c4bb985359f0faf71c320, State: Running, Role: LEADER
I20260925 22:48:51.428107 31682 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8f498fb67e5c4bb985359f0faf71c320 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f498fb67e5c4bb985359f0faf71c320" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45623 } }
I20260925 22:48:51.428574 31677 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8f498fb67e5c4bb985359f0faf71c320 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 22:48:51.428642 31680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:48:51.411738 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49845 (local address 127.0.0.1:45623)
0925 22:48:51.412315 (+ 577us) server_negotiation.cc:206] Beginning negotiation
0925 22:48:51.412339 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0925 22:48:51.412387 (+ 48us) server_negotiation.cc:407] Connection header received
0925 22:48:51.413327 (+ 940us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:48:51.413358 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:48:51.413875 (+ 517us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:48:51.414294 (+ 419us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:48:51.415124 (+ 830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:48:51.416911 (+ 1787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:48:51.418326 (+ 1415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:48:51.418918 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:48:51.419134 (+ 216us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:48:51.422089 (+ 2955us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:48:51.422096 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:48:51.422114 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:48:51.422158 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:48:51.424879 (+ 2721us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:48:51.425541 (+ 662us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:48:51.425548 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:48:51.425556 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:48:51.425705 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:48:51.426034 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:48:51.426038 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:48:51.426040 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:48:51.426575 (+ 535us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:48:51.426801 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0925 22:48:51.427161 (+ 360us) server_negotiation.cc:299] Negotiation successful
0925 22:48:51.427533 (+ 372us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":407,"thread_start_us":132,"threads_started":1}
I20260925 22:48:51.429450 31682 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8f498fb67e5c4bb985359f0faf71c320 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8f498fb67e5c4bb985359f0faf71c320" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f498fb67e5c4bb985359f0faf71c320" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45623 } } }
I20260925 22:48:51.429587 31682 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8f498fb67e5c4bb985359f0faf71c320 [sys.catalog]: This master's current role is: LEADER
I20260925 22:48:51.429795 31682 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8f498fb67e5c4bb985359f0faf71c320 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8f498fb67e5c4bb985359f0faf71c320. Latest consensus state: current_term: 1 leader_uuid: "8f498fb67e5c4bb985359f0faf71c320" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f498fb67e5c4bb985359f0faf71c320" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45623 } } }
I20260925 22:48:51.429970 31682 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8f498fb67e5c4bb985359f0faf71c320 [sys.catalog]: This master's current role is: LEADER
I20260925 22:48:51.430297 31691 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 22:48:51.432165 11032 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 22:48:51.434018 31691 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 22:48:51.441255 31691 catalog_manager.cc:1347] Generated new cluster ID: a7ac19a32c454fbd8db5d04a90d72916
I20260925 22:48:51.441399 31691 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 22:48:51.445883 31680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:48:51.434691 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43425 (local address 127.0.0.1:45623)
0925 22:48:51.434817 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0925 22:48:51.434823 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 22:48:51.436219 (+ 1396us) server_negotiation.cc:407] Connection header received
0925 22:48:51.436355 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:48:51.436359 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:48:51.436419 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:48:51.436526 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:48:51.437456 (+ 930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:48:51.438326 (+ 870us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:48:51.439469 (+ 1143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:48:51.439725 (+ 256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:48:51.439853 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:48:51.441734 (+ 1881us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:48:51.441738 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:48:51.441741 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:48:51.441773 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:48:51.443672 (+ 1899us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:48:51.444266 (+ 594us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:48:51.444270 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:48:51.444272 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:48:51.444373 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:48:51.444685 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:48:51.444689 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:48:51.444691 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:48:51.444956 (+ 265us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:48:51.445094 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0925 22:48:51.445356 (+ 262us) server_negotiation.cc:299] Negotiation successful
0925 22:48:51.445695 (+ 339us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260925 22:48:51.463095 31691 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 22:48:51.464246 31691 catalog_manager.cc:1504] Loading token signing keys...
I20260925 22:48:51.472414 31691 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8f498fb67e5c4bb985359f0faf71c320: Generated new TSK 0
I20260925 22:48:51.473445 31691 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 22:48:51.477761 11032 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrWXm8X/build/debug/bin/kudu with pid 31609
2026-09-25T22:48:51Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (1452 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790376259650642-11032-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 22:48:51 dist-test-slave-5vdr krb5kdc[31708](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 22:48:51 dist-test-slave-5vdr krb5kdc[31708](info): set up 2 sockets
Sep 25 22:48:51 dist-test-slave-5vdr krb5kdc[31708](info): commencing operation
krb5kdc: starting...
W20260925 22:48:52.528246 11032 mini_kdc.cc:121] Time spent starting KDC: real 1.030s user 0.006s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790376259650642-11032-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790376259650642-11032-0/krb5kdc/test-user.keytab.
Sep 25 22:48:52 dist-test-slave-5vdr krb5kdc[31708](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376532, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T22:48:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T22:48:52Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790376259650642-11032-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790376259650642-11032-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 22:48:52.660487 11032 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrWXm8X/build/debug/bin/kudu
/tmp/dist-test-taskrWXm8X/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790376259650642-11032-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790376259650642-11032-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790376259650642-11032-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790376259650642-11032-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33845
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790376259650642-11032-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:46587
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:33845
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790376259650642-11032-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790376259650642-11032-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790376259650642-11032-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:48:52.779825 31721 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:48:52.780224 31721 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:48:52.780314 31721 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:48:52.784703 31721 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 22:48:52.784838 31721 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:48:52.784880 31721 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:48:52.784919 31721 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 22:48:52.784953 31721 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 22:48:52.790993 31721 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:46587
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790376259650642-11032-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790376259650642-11032-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:33845
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790376259650642-11032-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33845
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790376259650642-11032-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31721
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790376259650642-11032-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:09:16 UTC on ccycloud.quasar-afmhyw.root.comops.site
build id 4673379
I20260925 22:48:52.792289 31721 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:48:52.793431 31721 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 22:48:52.802611 31721 server_base.cc:1034] running on GCE node
W20260925 22:48:52.802803 31729 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:48:52.802838 31727 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:48:52.802815 31726 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:48:52.803596 31721 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:48:52.804941 31721 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:48:52.806152 31721 hybrid_clock.cc:648] HybridClock initialized: now 1790376532806127 us; error 34 us; skew 500 ppm
Sep 25 22:48:52 dist-test-slave-5vdr krb5kdc[31708](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376532, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:48:52.810387 31721 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 22:48:52.811581 31721 webserver.cc:460] Webserver started at http://127.0.0.1:35293/ using document root <none> and password file <none>
I20260925 22:48:52.812331 31721 fs_manager.cc:359] Metadata directory not provided
I20260925 22:48:52.812439 31721 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:48:52.812682 31721 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:48:52.814991 31721 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790376259650642-11032-0/minicluster-data/master-0/data/instance:
uuid: "af3e41fd3a3749febf6306c08526189a"
format_stamp: "Formatted at 2026-09-25 22:48:52 on dist-test-slave-5vdr"
server_key: "a4224ab10d08dc071937e2a2e8919fb8"
server_key_iv: "1143fa4b044daf95c054e348e4c0f7f4"
server_key_version: "encryptionkey@0"
I20260925 22:48:52.815671 31721 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790376259650642-11032-0/minicluster-data/master-0/wal/instance:
uuid: "af3e41fd3a3749febf6306c08526189a"
format_stamp: "Formatted at 2026-09-25 22:48:52 on dist-test-slave-5vdr"
server_key: "a4224ab10d08dc071937e2a2e8919fb8"
server_key_iv: "1143fa4b044daf95c054e348e4c0f7f4"
server_key_version: "encryptionkey@0"
I20260925 22:48:52.820056 31721 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260925 22:48:52.822971 31736 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:48:52.824507 31721 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 22:48:52.824712 31721 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790376259650642-11032-0/minicluster-data/master-0/data,/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790376259650642-11032-0/minicluster-data/master-0/wal
uuid: "af3e41fd3a3749febf6306c08526189a"
format_stamp: "Formatted at 2026-09-25 22:48:52 on dist-test-slave-5vdr"
server_key: "a4224ab10d08dc071937e2a2e8919fb8"
server_key_iv: "1143fa4b044daf95c054e348e4c0f7f4"
server_key_version: "encryptionkey@0"
I20260925 22:48:52.824870 31721 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790376259650642-11032-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790376259650642-11032-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790376259650642-11032-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:48:52.841464 31721 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:48:52.845402 31721 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:48:52.845703 31721 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:48:52.854200 31721 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:33845
I20260925 22:48:52.854206 31788 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:33845 every 8 connection(s)
I20260925 22:48:52.855444 31721 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790376259650642-11032-0/minicluster-data/master-0/data/info.pb
I20260925 22:48:52.858353 11032 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrWXm8X/build/debug/bin/kudu as pid 31721
I20260925 22:48:52.858517 11032 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790376259650642-11032-0/minicluster-data/master-0/wal/instance
I20260925 22:48:52.858841 11032 external_mini_cluster.cc:1428] Setting key 8e08609b2722f62d331dc888c2bbb592
I20260925 22:48:52.858989 31789 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:48:52.867030 31789 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P af3e41fd3a3749febf6306c08526189a: Bootstrap starting.
I20260925 22:48:52.869210 31792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:48:52.860371 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56221 (local address 127.0.0.1:33845)
0925 22:48:52.860866 (+ 495us) server_negotiation.cc:206] Beginning negotiation
0925 22:48:52.860884 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0925 22:48:52.860925 (+ 41us) server_negotiation.cc:407] Connection header received
0925 22:48:52.861889 (+ 964us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:48:52.861913 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:48:52.862305 (+ 392us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:48:52.862622 (+ 317us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:48:52.863061 (+ 439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:48:52.864227 (+ 1166us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:48:52.865539 (+ 1312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:48:52.866088 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:48:52.866234 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:48:52.867258 (+ 1024us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:48:52.867265 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:48:52.867280 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 22:48:52.867724 (+ 444us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:48:52.867788 (+ 64us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 22:48:52.867814 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:48:52.867975 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0925 22:48:52.868124 (+ 149us) server_negotiation.cc:299] Negotiation successful
0925 22:48:52.868386 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":329,"thread_start_us":182,"threads_started":1}
I20260925 22:48:52.869789 31789 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P af3e41fd3a3749febf6306c08526189a: Neither blocks nor log segments found. Creating new log.
I20260925 22:48:52.870710 31789 log.cc:824] T 00000000000000000000000000000000 P af3e41fd3a3749febf6306c08526189a: Log is configured to *not* fsync() on all Append() calls
I20260925 22:48:52.872728 11032 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 22:48:52.873361 31789 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P af3e41fd3a3749febf6306c08526189a: No bootstrap required, opened a new log
I20260925 22:48:52.877374 31789 raft_consensus.cc:348] T 00000000000000000000000000000000 P af3e41fd3a3749febf6306c08526189a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af3e41fd3a3749febf6306c08526189a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33845 } }
I20260925 22:48:52.877732 31789 raft_consensus.cc:374] T 00000000000000000000000000000000 P af3e41fd3a3749febf6306c08526189a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:48:52.877799 31789 raft_consensus.cc:729] T 00000000000000000000000000000000 P af3e41fd3a3749febf6306c08526189a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af3e41fd3a3749febf6306c08526189a, State: Initialized, Role: FOLLOWER
I20260925 22:48:52.878325 31789 consensus_queue.cc:260] T 00000000000000000000000000000000 P af3e41fd3a3749febf6306c08526189a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af3e41fd3a3749febf6306c08526189a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33845 } }
I20260925 22:48:52.878540 31789 raft_consensus.cc:388] T 00000000000000000000000000000000 P af3e41fd3a3749febf6306c08526189a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 22:48:52.878624 31789 raft_consensus.cc:482] T 00000000000000000000000000000000 P af3e41fd3a3749febf6306c08526189a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 22:48:52.878751 31789 raft_consensus.cc:3037] T 00000000000000000000000000000000 P af3e41fd3a3749febf6306c08526189a [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:48:52.879798 31792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:48:52.874960 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48329 (local address 127.0.0.1:33845)
0925 22:48:52.875103 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0925 22:48:52.875107 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:48:52.875263 (+ 156us) server_negotiation.cc:407] Connection header received
0925 22:48:52.875401 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:48:52.875403 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:48:52.875445 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:48:52.875516 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:48:52.875907 (+ 391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:48:52.876792 (+ 885us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:48:52.878007 (+ 1215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:48:52.878316 (+ 309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:48:52.878409 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:48:52.879355 (+ 946us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:48:52.879359 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:48:52.879361 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 22:48:52.879374 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:48:52.879401 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 22:48:52.879412 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:48:52.879475 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0925 22:48:52.879602 (+ 127us) server_negotiation.cc:299] Negotiation successful
0925 22:48:52.879654 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260925 22:48:52.880257 31789 raft_consensus.cc:504] T 00000000000000000000000000000000 P af3e41fd3a3749febf6306c08526189a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af3e41fd3a3749febf6306c08526189a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33845 } }
I20260925 22:48:52.880664 31789 leader_election.cc:302] T 00000000000000000000000000000000 P af3e41fd3a3749febf6306c08526189a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: af3e41fd3a3749febf6306c08526189a; no voters:
I20260925 22:48:52.881062 31789 leader_election.cc:288] T 00000000000000000000000000000000 P af3e41fd3a3749febf6306c08526189a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 22:48:52.881212 31799 raft_consensus.cc:2781] T 00000000000000000000000000000000 P af3e41fd3a3749febf6306c08526189a [term 1 FOLLOWER]: Leader election won for term 1
I20260925 22:48:52.881464 31799 raft_consensus.cc:686] T 00000000000000000000000000000000 P af3e41fd3a3749febf6306c08526189a [term 1 LEADER]: Becoming Leader. State: Replica: af3e41fd3a3749febf6306c08526189a, State: Running, Role: LEADER
I20260925 22:48:52.881855 31799 consensus_queue.cc:237] T 00000000000000000000000000000000 P af3e41fd3a3749febf6306c08526189a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af3e41fd3a3749febf6306c08526189a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33845 } }
I20260925 22:48:52.882468 31789 sys_catalog.cc:564] T 00000000000000000000000000000000 P af3e41fd3a3749febf6306c08526189a [sys.catalog]: configured and running, proceeding with master startup.
I20260925 22:48:52.883703 31801 sys_catalog.cc:455] T 00000000000000000000000000000000 P af3e41fd3a3749febf6306c08526189a [sys.catalog]: SysCatalogTable state changed. Reason: New leader af3e41fd3a3749febf6306c08526189a. Latest consensus state: current_term: 1 leader_uuid: "af3e41fd3a3749febf6306c08526189a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af3e41fd3a3749febf6306c08526189a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33845 } } }
I20260925 22:48:52.883925 31801 sys_catalog.cc:458] T 00000000000000000000000000000000 P af3e41fd3a3749febf6306c08526189a [sys.catalog]: This master's current role is: LEADER
I20260925 22:48:52.884330 31800 sys_catalog.cc:455] T 00000000000000000000000000000000 P af3e41fd3a3749febf6306c08526189a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "af3e41fd3a3749febf6306c08526189a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af3e41fd3a3749febf6306c08526189a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33845 } } }
I20260925 22:48:52.884608 31800 sys_catalog.cc:458] T 00000000000000000000000000000000 P af3e41fd3a3749febf6306c08526189a [sys.catalog]: This master's current role is: LEADER
W20260925 22:48:52.887735 31814 catalog_manager.cc:1558] T 00000000000000000000000000000000 P af3e41fd3a3749febf6306c08526189a: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 22:48:52.887835 31814 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 22:48:52.887992 31808 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 22:48:52.889011 31808 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 22:48:52.896154 31808 catalog_manager.cc:1347] Generated new cluster ID: c955791572854565ad562d7b9269f3b3
I20260925 22:48:52.896270 31808 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 22:48:52.907872 31808 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 22:48:52.909022 31808 catalog_manager.cc:1504] Loading token signing keys...
I20260925 22:48:52.917057 31808 catalog_manager.cc:5868] T 00000000000000000000000000000000 P af3e41fd3a3749febf6306c08526189a: Generated new TSK 0
I20260925 22:48:52.918030 31808 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 22:48:52.929482 11032 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrWXm8X/build/debug/bin/kudu with pid 31721
2026-09-25T22:48:52Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1451 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790376259650642-11032-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 22:48:52 dist-test-slave-5vdr krb5kdc[31820](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 22:48:52 dist-test-slave-5vdr krb5kdc[31820](info): set up 2 sockets
Sep 25 22:48:52 dist-test-slave-5vdr krb5kdc[31820](info): commencing operation
krb5kdc: starting...
W20260925 22:48:53.984261 11032 mini_kdc.cc:121] Time spent starting KDC: real 1.034s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790376259650642-11032-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790376259650642-11032-0/krb5kdc/test-user.keytab.
Sep 25 22:48:54 dist-test-slave-5vdr krb5kdc[31820](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376534, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T22:48:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T22:48:54Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790376259650642-11032-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790376259650642-11032-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 22:48:54.113263 11032 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrWXm8X/build/debug/bin/kudu
/tmp/dist-test-taskrWXm8X/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790376259650642-11032-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790376259650642-11032-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790376259650642-11032-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790376259650642-11032-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39359
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790376259650642-11032-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40537
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39359
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790376259650642-11032-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790376259650642-11032-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790376259650642-11032-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:48:54.230926 31833 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:48:54.231302 31833 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:48:54.231382 31833 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:48:54.235785 31833 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 22:48:54.235970 31833 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:48:54.236009 31833 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:48:54.236052 31833 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 22:48:54.236086 31833 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 22:48:54.242013 31833 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40537
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790376259650642-11032-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790376259650642-11032-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39359
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790376259650642-11032-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39359
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790376259650642-11032-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31833
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790376259650642-11032-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:09:16 UTC on ccycloud.quasar-afmhyw.root.comops.site
build id 4673379
I20260925 22:48:54.243286 31833 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:48:54.244510 31833 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:48:54.253978 31839 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:48:54.253993 31841 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:48:54.253979 31838 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:48:54.254416 31833 server_base.cc:1034] running on GCE node
I20260925 22:48:54.255139 31833 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:48:54.256680 31833 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:48:54.257886 31833 hybrid_clock.cc:648] HybridClock initialized: now 1790376534257862 us; error 65 us; skew 500 ppm
Sep 25 22:48:54 dist-test-slave-5vdr krb5kdc[31820](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376534, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:48:54.261379 31833 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 22:48:54.262635 31833 webserver.cc:460] Webserver started at http://127.0.0.1:44983/ using document root <none> and password file <none>
I20260925 22:48:54.263309 31833 fs_manager.cc:359] Metadata directory not provided
I20260925 22:48:54.263412 31833 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:48:54.263690 31833 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:48:54.266096 31833 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790376259650642-11032-0/minicluster-data/master-0/data/instance:
uuid: "687869af69a9490fa7ad4fc46246afed"
format_stamp: "Formatted at 2026-09-25 22:48:54 on dist-test-slave-5vdr"
server_key: "bfad2395d199b8bad59fff966f529e6e"
server_key_iv: "3315dc6d23491c0a3e1eb94fe37b6c62"
server_key_version: "encryptionkey@0"
I20260925 22:48:54.266744 31833 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790376259650642-11032-0/minicluster-data/master-0/wal/instance:
uuid: "687869af69a9490fa7ad4fc46246afed"
format_stamp: "Formatted at 2026-09-25 22:48:54 on dist-test-slave-5vdr"
server_key: "bfad2395d199b8bad59fff966f529e6e"
server_key_iv: "3315dc6d23491c0a3e1eb94fe37b6c62"
server_key_version: "encryptionkey@0"
I20260925 22:48:54.271159 31833 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260925 22:48:54.274233 31848 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:48:54.275635 31833 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260925 22:48:54.275825 31833 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790376259650642-11032-0/minicluster-data/master-0/data,/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790376259650642-11032-0/minicluster-data/master-0/wal
uuid: "687869af69a9490fa7ad4fc46246afed"
format_stamp: "Formatted at 2026-09-25 22:48:54 on dist-test-slave-5vdr"
server_key: "bfad2395d199b8bad59fff966f529e6e"
server_key_iv: "3315dc6d23491c0a3e1eb94fe37b6c62"
server_key_version: "encryptionkey@0"
I20260925 22:48:54.276046 31833 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790376259650642-11032-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790376259650642-11032-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790376259650642-11032-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:48:54.281450 31833 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:48:54.284897 31833 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:48:54.285142 31833 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:48:54.292886 31833 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39359
I20260925 22:48:54.292907 31900 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39359 every 8 connection(s)
I20260925 22:48:54.293993 31833 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790376259650642-11032-0/minicluster-data/master-0/data/info.pb
I20260925 22:48:54.297261 31901 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:48:54.300758 11032 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrWXm8X/build/debug/bin/kudu as pid 31833
I20260925 22:48:54.300944 11032 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790376259650642-11032-0/minicluster-data/master-0/wal/instance
I20260925 22:48:54.301232 11032 external_mini_cluster.cc:1428] Setting key 958709bffbb39290ffb5d5bc4578b444
I20260925 22:48:54.303922 31901 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 687869af69a9490fa7ad4fc46246afed: Bootstrap starting.
I20260925 22:48:54.306814 31901 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 687869af69a9490fa7ad4fc46246afed: Neither blocks nor log segments found. Creating new log.
I20260925 22:48:54.307477 31904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:48:54.302853 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52977 (local address 127.0.0.1:39359)
0925 22:48:54.303363 (+ 510us) server_negotiation.cc:206] Beginning negotiation
0925 22:48:54.303381 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0925 22:48:54.303423 (+ 42us) server_negotiation.cc:407] Connection header received
0925 22:48:54.304285 (+ 862us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:48:54.304305 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:48:54.304637 (+ 332us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:48:54.304972 (+ 335us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:48:54.305960 (+ 988us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:48:54.305965 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:48:54.305978 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 22:48:54.305990 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:48:54.306052 (+ 62us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 22:48:54.306075 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:48:54.306179 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0925 22:48:54.306330 (+ 151us) server_negotiation.cc:299] Negotiation successful
0925 22:48:54.306539 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":333,"thread_start_us":178,"threads_started":1}
I20260925 22:48:54.307750 31901 log.cc:824] T 00000000000000000000000000000000 P 687869af69a9490fa7ad4fc46246afed: Log is configured to *not* fsync() on all Append() calls
I20260925 22:48:54.310176 31901 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 687869af69a9490fa7ad4fc46246afed: No bootstrap required, opened a new log
I20260925 22:48:54.310667 11032 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 22:48:54.315021 31904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:48:54.312922 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46125 (local address 127.0.0.1:39359)
0925 22:48:54.313077 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0925 22:48:54.313084 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 22:48:54.313206 (+ 122us) server_negotiation.cc:407] Connection header received
0925 22:48:54.313327 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:48:54.313333 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:48:54.313400 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:48:54.313545 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:48:54.314412 (+ 867us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:48:54.314416 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:48:54.314419 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 22:48:54.314525 (+ 106us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:48:54.314580 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 22:48:54.314602 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:48:54.314715 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0925 22:48:54.314826 (+ 111us) server_negotiation.cc:299] Negotiation successful
0925 22:48:54.314874 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260925 22:48:54.315248 31901 raft_consensus.cc:348] T 00000000000000000000000000000000 P 687869af69a9490fa7ad4fc46246afed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "687869af69a9490fa7ad4fc46246afed" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39359 } }
I20260925 22:48:54.315630 31901 raft_consensus.cc:374] T 00000000000000000000000000000000 P 687869af69a9490fa7ad4fc46246afed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:48:54.315696 31901 raft_consensus.cc:729] T 00000000000000000000000000000000 P 687869af69a9490fa7ad4fc46246afed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 687869af69a9490fa7ad4fc46246afed, State: Initialized, Role: FOLLOWER
I20260925 22:48:54.316301 31901 consensus_queue.cc:260] T 00000000000000000000000000000000 P 687869af69a9490fa7ad4fc46246afed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "687869af69a9490fa7ad4fc46246afed" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39359 } }
I20260925 22:48:54.316484 31901 raft_consensus.cc:388] T 00000000000000000000000000000000 P 687869af69a9490fa7ad4fc46246afed [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 22:48:54.316555 31901 raft_consensus.cc:482] T 00000000000000000000000000000000 P 687869af69a9490fa7ad4fc46246afed [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 22:48:54.316659 31901 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 687869af69a9490fa7ad4fc46246afed [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:48:54.317739 31901 raft_consensus.cc:504] T 00000000000000000000000000000000 P 687869af69a9490fa7ad4fc46246afed [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "687869af69a9490fa7ad4fc46246afed" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39359 } }
I20260925 22:48:54.318252 31901 leader_election.cc:302] T 00000000000000000000000000000000 P 687869af69a9490fa7ad4fc46246afed [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 687869af69a9490fa7ad4fc46246afed; no voters:
I20260925 22:48:54.318643 31901 leader_election.cc:288] T 00000000000000000000000000000000 P 687869af69a9490fa7ad4fc46246afed [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 22:48:54.318792 31911 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 687869af69a9490fa7ad4fc46246afed [term 1 FOLLOWER]: Leader election won for term 1
I20260925 22:48:54.319072 31911 raft_consensus.cc:686] T 00000000000000000000000000000000 P 687869af69a9490fa7ad4fc46246afed [term 1 LEADER]: Becoming Leader. State: Replica: 687869af69a9490fa7ad4fc46246afed, State: Running, Role: LEADER
I20260925 22:48:54.319463 31911 consensus_queue.cc:237] T 00000000000000000000000000000000 P 687869af69a9490fa7ad4fc46246afed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "687869af69a9490fa7ad4fc46246afed" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39359 } }
I20260925 22:48:54.320137 31901 sys_catalog.cc:564] T 00000000000000000000000000000000 P 687869af69a9490fa7ad4fc46246afed [sys.catalog]: configured and running, proceeding with master startup.
I20260925 22:48:54.321442 31912 sys_catalog.cc:455] T 00000000000000000000000000000000 P 687869af69a9490fa7ad4fc46246afed [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "687869af69a9490fa7ad4fc46246afed" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "687869af69a9490fa7ad4fc46246afed" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39359 } } }
I20260925 22:48:54.321607 31912 sys_catalog.cc:458] T 00000000000000000000000000000000 P 687869af69a9490fa7ad4fc46246afed [sys.catalog]: This master's current role is: LEADER
I20260925 22:48:54.321961 31913 sys_catalog.cc:455] T 00000000000000000000000000000000 P 687869af69a9490fa7ad4fc46246afed [sys.catalog]: SysCatalogTable state changed. Reason: New leader 687869af69a9490fa7ad4fc46246afed. Latest consensus state: current_term: 1 leader_uuid: "687869af69a9490fa7ad4fc46246afed" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "687869af69a9490fa7ad4fc46246afed" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39359 } } }
I20260925 22:48:54.322064 31920 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 22:48:54.323378 31913 sys_catalog.cc:458] T 00000000000000000000000000000000 P 687869af69a9490fa7ad4fc46246afed [sys.catalog]: This master's current role is: LEADER
I20260925 22:48:54.325340 31920 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 22:48:54.332201 31920 catalog_manager.cc:1347] Generated new cluster ID: a56a0a8c49d7462a90ece3d5bbf16a8f
I20260925 22:48:54.332324 31920 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 22:48:54.345002 31920 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 22:48:54.345381 31920 catalog_manager.cc:1504] Loading token signing keys...
I20260925 22:48:54.351898 31920 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 687869af69a9490fa7ad4fc46246afed: Generated new TSK 0
I20260925 22:48:54.352728 31920 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 22:48:54.360994 11032 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrWXm8X/build/debug/bin/kudu with pid 31833
2026-09-25T22:48:54Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1432 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790376259650642-11032-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 22:48:54 dist-test-slave-5vdr krb5kdc[31932](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 22:48:54 dist-test-slave-5vdr krb5kdc[31932](info): set up 2 sockets
Sep 25 22:48:54 dist-test-slave-5vdr krb5kdc[31932](info): commencing operation
krb5kdc: starting...
W20260925 22:48:55.416350 11032 mini_kdc.cc:121] Time spent starting KDC: real 1.034s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790376259650642-11032-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790376259650642-11032-0/krb5kdc/test-user.keytab.
Sep 25 22:48:55 dist-test-slave-5vdr krb5kdc[31932](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376535, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T22:48:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T22:48:55Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790376259650642-11032-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790376259650642-11032-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 22:48:55.556763 11032 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrWXm8X/build/debug/bin/kudu
/tmp/dist-test-taskrWXm8X/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790376259650642-11032-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790376259650642-11032-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790376259650642-11032-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790376259650642-11032-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42679
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790376259650642-11032-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39953
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42679
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790376259650642-11032-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790376259650642-11032-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790376259650642-11032-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:48:55.676872 31945 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:48:55.677206 31945 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:48:55.677253 31945 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:48:55.681700 31945 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 22:48:55.681789 31945 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:48:55.681810 31945 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:48:55.681828 31945 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 22:48:55.681843 31945 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 22:48:55.687911 31945 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39953
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790376259650642-11032-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790376259650642-11032-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42679
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790376259650642-11032-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42679
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790376259650642-11032-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31945
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790376259650642-11032-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:09:16 UTC on ccycloud.quasar-afmhyw.root.comops.site
build id 4673379
I20260925 22:48:55.689168 31945 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:48:55.690255 31945 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:48:55.699741 31950 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:48:55.699833 31945 server_base.cc:1034] running on GCE node
W20260925 22:48:55.699746 31953 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:48:55.699790 31951 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:48:55.700802 31945 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:48:55.702080 31945 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:48:55.703274 31945 hybrid_clock.cc:648] HybridClock initialized: now 1790376535703247 us; error 44 us; skew 500 ppm
Sep 25 22:48:55 dist-test-slave-5vdr krb5kdc[31932](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376535, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:48:55.706555 31945 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 22:48:55.707615 31945 webserver.cc:460] Webserver started at http://127.0.0.1:44473/ using document root <none> and password file <none>
I20260925 22:48:55.708293 31945 fs_manager.cc:359] Metadata directory not provided
I20260925 22:48:55.708367 31945 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:48:55.708618 31945 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:48:55.710820 31945 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790376259650642-11032-0/minicluster-data/master-0/data/instance:
uuid: "cfd5d7276f344d2e9066f67c7fbb6d32"
format_stamp: "Formatted at 2026-09-25 22:48:55 on dist-test-slave-5vdr"
server_key: "7f01bbea649463a3e9308efdcd16bb48"
server_key_iv: "37b6590b5b6ff1710cf54c6ef751e57e"
server_key_version: "encryptionkey@0"
I20260925 22:48:55.711472 31945 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790376259650642-11032-0/minicluster-data/master-0/wal/instance:
uuid: "cfd5d7276f344d2e9066f67c7fbb6d32"
format_stamp: "Formatted at 2026-09-25 22:48:55 on dist-test-slave-5vdr"
server_key: "7f01bbea649463a3e9308efdcd16bb48"
server_key_iv: "37b6590b5b6ff1710cf54c6ef751e57e"
server_key_version: "encryptionkey@0"
I20260925 22:48:55.715675 31945 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260925 22:48:55.718714 31960 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:48:55.720343 31945 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 22:48:55.720541 31945 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790376259650642-11032-0/minicluster-data/master-0/data,/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790376259650642-11032-0/minicluster-data/master-0/wal
uuid: "cfd5d7276f344d2e9066f67c7fbb6d32"
format_stamp: "Formatted at 2026-09-25 22:48:55 on dist-test-slave-5vdr"
server_key: "7f01bbea649463a3e9308efdcd16bb48"
server_key_iv: "37b6590b5b6ff1710cf54c6ef751e57e"
server_key_version: "encryptionkey@0"
I20260925 22:48:55.720687 31945 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790376259650642-11032-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790376259650642-11032-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790376259650642-11032-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:48:55.737427 31945 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:48:55.741223 31945 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:48:55.741479 31945 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:48:55.749781 31945 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42679
I20260925 22:48:55.749830 32012 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42679 every 8 connection(s)
I20260925 22:48:55.750887 31945 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790376259650642-11032-0/minicluster-data/master-0/data/info.pb
I20260925 22:48:55.754477 11032 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrWXm8X/build/debug/bin/kudu as pid 31945
I20260925 22:48:55.754645 11032 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790376259650642-11032-0/minicluster-data/master-0/wal/instance
I20260925 22:48:55.754694 32013 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:48:55.754963 11032 external_mini_cluster.cc:1428] Setting key 552b91c04ebe4989c31aa4d7e73c9162
I20260925 22:48:55.762653 32013 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cfd5d7276f344d2e9066f67c7fbb6d32: Bootstrap starting.
Sep 25 22:48:55 dist-test-slave-5vdr krb5kdc[31932](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376535, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260925 22:48:55.765887 32013 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cfd5d7276f344d2e9066f67c7fbb6d32: Neither blocks nor log segments found. Creating new log.
I20260925 22:48:55.766820 32013 log.cc:824] T 00000000000000000000000000000000 P cfd5d7276f344d2e9066f67c7fbb6d32: Log is configured to *not* fsync() on all Append() calls
I20260925 22:48:55.769327 32013 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cfd5d7276f344d2e9066f67c7fbb6d32: No bootstrap required, opened a new log
I20260925 22:48:55.771466 32016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:48:55.756467 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.97:40695 (local address 127.0.0.1:42679)
0925 22:48:55.756971 (+ 504us) server_negotiation.cc:206] Beginning negotiation
0925 22:48:55.756988 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0925 22:48:55.757030 (+ 42us) server_negotiation.cc:407] Connection header received
0925 22:48:55.757714 (+ 684us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:48:55.757734 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:48:55.758162 (+ 428us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:48:55.758559 (+ 397us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:48:55.759652 (+ 1093us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:48:55.760828 (+ 1176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:48:55.761788 (+ 960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:48:55.762343 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:48:55.762556 (+ 213us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:48:55.765874 (+ 3318us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:48:55.765888 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:48:55.765906 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:48:55.765963 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:48:55.768195 (+ 2232us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:48:55.768848 (+ 653us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:48:55.768855 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:48:55.768861 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:48:55.768980 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:48:55.769352 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:48:55.769356 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:48:55.769358 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:48:55.769845 (+ 487us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:48:55.770086 (+ 241us) server_negotiation.cc:1035] Waiting for connection context
0925 22:48:55.770431 (+ 345us) server_negotiation.cc:299] Negotiation successful
0925 22:48:55.770682 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":341,"thread_start_us":153,"threads_started":1}
I20260925 22:48:55.773749 32013 raft_consensus.cc:348] T 00000000000000000000000000000000 P cfd5d7276f344d2e9066f67c7fbb6d32 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cfd5d7276f344d2e9066f67c7fbb6d32" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42679 } }
I20260925 22:48:55.774102 32013 raft_consensus.cc:374] T 00000000000000000000000000000000 P cfd5d7276f344d2e9066f67c7fbb6d32 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:48:55.774217 32013 raft_consensus.cc:729] T 00000000000000000000000000000000 P cfd5d7276f344d2e9066f67c7fbb6d32 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cfd5d7276f344d2e9066f67c7fbb6d32, State: Initialized, Role: FOLLOWER
I20260925 22:48:55.774784 32013 consensus_queue.cc:260] T 00000000000000000000000000000000 P cfd5d7276f344d2e9066f67c7fbb6d32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cfd5d7276f344d2e9066f67c7fbb6d32" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42679 } }
I20260925 22:48:55.774899 11032 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 22:48:55.775012 32013 raft_consensus.cc:388] T 00000000000000000000000000000000 P cfd5d7276f344d2e9066f67c7fbb6d32 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 22:48:55.775095 32013 raft_consensus.cc:482] T 00000000000000000000000000000000 P cfd5d7276f344d2e9066f67c7fbb6d32 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 22:48:55.775211 32013 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cfd5d7276f344d2e9066f67c7fbb6d32 [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:48:55.776582 32013 raft_consensus.cc:504] T 00000000000000000000000000000000 P cfd5d7276f344d2e9066f67c7fbb6d32 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cfd5d7276f344d2e9066f67c7fbb6d32" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42679 } }
I20260925 22:48:55.777024 32013 leader_election.cc:302] T 00000000000000000000000000000000 P cfd5d7276f344d2e9066f67c7fbb6d32 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cfd5d7276f344d2e9066f67c7fbb6d32; no voters:
I20260925 22:48:55.777454 32013 leader_election.cc:288] T 00000000000000000000000000000000 P cfd5d7276f344d2e9066f67c7fbb6d32 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 22:48:55.777937 32024 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cfd5d7276f344d2e9066f67c7fbb6d32 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 22:48:55.778224 32024 raft_consensus.cc:686] T 00000000000000000000000000000000 P cfd5d7276f344d2e9066f67c7fbb6d32 [term 1 LEADER]: Becoming Leader. State: Replica: cfd5d7276f344d2e9066f67c7fbb6d32, State: Running, Role: LEADER
I20260925 22:48:55.778599 32024 consensus_queue.cc:237] T 00000000000000000000000000000000 P cfd5d7276f344d2e9066f67c7fbb6d32 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cfd5d7276f344d2e9066f67c7fbb6d32" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42679 } }
I20260925 22:48:55.779210 32013 sys_catalog.cc:564] T 00000000000000000000000000000000 P cfd5d7276f344d2e9066f67c7fbb6d32 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 22:48:55.780313 32023 sys_catalog.cc:455] T 00000000000000000000000000000000 P cfd5d7276f344d2e9066f67c7fbb6d32 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cfd5d7276f344d2e9066f67c7fbb6d32" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cfd5d7276f344d2e9066f67c7fbb6d32" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42679 } } }
I20260925 22:48:55.780457 32023 sys_catalog.cc:458] T 00000000000000000000000000000000 P cfd5d7276f344d2e9066f67c7fbb6d32 [sys.catalog]: This master's current role is: LEADER
I20260925 22:48:55.780802 32025 sys_catalog.cc:455] T 00000000000000000000000000000000 P cfd5d7276f344d2e9066f67c7fbb6d32 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cfd5d7276f344d2e9066f67c7fbb6d32. Latest consensus state: current_term: 1 leader_uuid: "cfd5d7276f344d2e9066f67c7fbb6d32" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cfd5d7276f344d2e9066f67c7fbb6d32" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42679 } } }
I20260925 22:48:55.780900 32025 sys_catalog.cc:458] T 00000000000000000000000000000000 P cfd5d7276f344d2e9066f67c7fbb6d32 [sys.catalog]: This master's current role is: LEADER
W20260925 22:48:55.785235 32038 catalog_manager.cc:1558] T 00000000000000000000000000000000 P cfd5d7276f344d2e9066f67c7fbb6d32: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 22:48:55.785346 32038 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 22:48:55.785893 32032 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 22:48:55.786788 32032 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 22:48:55.788463 32016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:48:55.777327 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.97:44341 (local address 127.0.0.1:42679)
0925 22:48:55.777502 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0925 22:48:55.777508 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 22:48:55.777678 (+ 170us) server_negotiation.cc:407] Connection header received
0925 22:48:55.777796 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:48:55.777799 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:48:55.777861 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:48:55.777960 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:48:55.779026 (+ 1066us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:48:55.779921 (+ 895us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:48:55.781434 (+ 1513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:48:55.781785 (+ 351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:48:55.781897 (+ 112us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:48:55.783827 (+ 1930us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:48:55.783832 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:48:55.783835 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:48:55.783908 (+ 73us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:48:55.785738 (+ 1830us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:48:55.786427 (+ 689us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:48:55.786431 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:48:55.786433 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:48:55.786536 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:48:55.786951 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:48:55.786954 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:48:55.786956 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:48:55.787211 (+ 255us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:48:55.787360 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0925 22:48:55.788187 (+ 827us) server_negotiation.cc:299] Negotiation successful
0925 22:48:55.788300 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260925 22:48:55.793699 32032 catalog_manager.cc:1347] Generated new cluster ID: 039e1e5fa57b469d96e4943b4d7db867
I20260925 22:48:55.793803 32032 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 22:48:55.804191 32032 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 22:48:55.805280 32032 catalog_manager.cc:1504] Loading token signing keys...
I20260925 22:48:55.811617 32032 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cfd5d7276f344d2e9066f67c7fbb6d32: Generated new TSK 0
I20260925 22:48:55.812469 32032 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 22:48:55.822170 11032 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrWXm8X/build/debug/bin/kudu with pid 31945
W20260925 22:48:55.828356 31930 connection.cc:535] client connection to 127.0.0.1:42679 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:42679): Connection reset by peer (error 104)
2026-09-25T22:48:55Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (1461 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790376259650642-11032-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 22:48:55 dist-test-slave-5vdr krb5kdc[32044](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 22:48:55 dist-test-slave-5vdr krb5kdc[32044](info): set up 2 sockets
Sep 25 22:48:55 dist-test-slave-5vdr krb5kdc[32044](info): commencing operation
krb5kdc: starting...
W20260925 22:48:56.884344 11032 mini_kdc.cc:121] Time spent starting KDC: real 1.041s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790376259650642-11032-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790376259650642-11032-0/krb5kdc/test-user.keytab.
Sep 25 22:48:56 dist-test-slave-5vdr krb5kdc[32044](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376536, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T22:48:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T22:48:56Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790376259650642-11032-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790376259650642-11032-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 22:48:57.016146 11032 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrWXm8X/build/debug/bin/kudu
/tmp/dist-test-taskrWXm8X/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790376259650642-11032-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790376259650642-11032-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790376259650642-11032-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790376259650642-11032-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34453
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790376259650642-11032-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39379
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34453
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790376259650642-11032-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790376259650642-11032-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790376259650642-11032-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:48:57.140340 32057 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:48:57.140744 32057 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:48:57.140831 32057 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:48:57.145458 32057 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 22:48:57.145583 32057 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:48:57.145619 32057 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:48:57.145663 32057 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 22:48:57.145694 32057 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 22:48:57.151670 32057 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39379
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790376259650642-11032-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790376259650642-11032-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34453
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790376259650642-11032-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34453
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790376259650642-11032-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32057
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790376259650642-11032-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:09:16 UTC on ccycloud.quasar-afmhyw.root.comops.site
build id 4673379
I20260925 22:48:57.152997 32057 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:48:57.154153 32057 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:48:57.163949 32065 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:48:57.164037 32062 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:48:57.163949 32063 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:48:57.164213 32057 server_base.cc:1034] running on GCE node
I20260925 22:48:57.165028 32057 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:48:57.166463 32057 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:48:57.167723 32057 hybrid_clock.cc:648] HybridClock initialized: now 1790376537167708 us; error 47 us; skew 500 ppm
Sep 25 22:48:57 dist-test-slave-5vdr krb5kdc[32044](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376537, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:48:57.171186 32057 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 22:48:57.172406 32057 webserver.cc:460] Webserver started at http://127.0.0.1:41055/ using document root <none> and password file <none>
I20260925 22:48:57.173110 32057 fs_manager.cc:359] Metadata directory not provided
I20260925 22:48:57.173261 32057 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:48:57.173539 32057 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:48:57.176107 32057 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790376259650642-11032-0/minicluster-data/master-0/data/instance:
uuid: "30f858a2009643eca6321f8b49c902ba"
format_stamp: "Formatted at 2026-09-25 22:48:57 on dist-test-slave-5vdr"
server_key: "4e8cc8be1b966c3a4cddc75c2bba0816"
server_key_iv: "68fcf8e9c9b4b5ba6d58dd5eb454fe49"
server_key_version: "encryptionkey@0"
I20260925 22:48:57.176815 32057 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790376259650642-11032-0/minicluster-data/master-0/wal/instance:
uuid: "30f858a2009643eca6321f8b49c902ba"
format_stamp: "Formatted at 2026-09-25 22:48:57 on dist-test-slave-5vdr"
server_key: "4e8cc8be1b966c3a4cddc75c2bba0816"
server_key_iv: "68fcf8e9c9b4b5ba6d58dd5eb454fe49"
server_key_version: "encryptionkey@0"
I20260925 22:48:57.181308 32057 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.003s
I20260925 22:48:57.184489 32072 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:48:57.186242 32057 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260925 22:48:57.186467 32057 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790376259650642-11032-0/minicluster-data/master-0/data,/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790376259650642-11032-0/minicluster-data/master-0/wal
uuid: "30f858a2009643eca6321f8b49c902ba"
format_stamp: "Formatted at 2026-09-25 22:48:57 on dist-test-slave-5vdr"
server_key: "4e8cc8be1b966c3a4cddc75c2bba0816"
server_key_iv: "68fcf8e9c9b4b5ba6d58dd5eb454fe49"
server_key_version: "encryptionkey@0"
I20260925 22:48:57.186635 32057 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790376259650642-11032-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790376259650642-11032-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790376259650642-11032-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:48:57.209761 32057 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:48:57.213943 32057 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:48:57.214218 32057 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:48:57.222997 32057 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34453
I20260925 22:48:57.223027 32124 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34453 every 8 connection(s)
I20260925 22:48:57.224324 32057 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790376259650642-11032-0/minicluster-data/master-0/data/info.pb
I20260925 22:48:57.227244 11032 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrWXm8X/build/debug/bin/kudu as pid 32057
I20260925 22:48:57.227420 11032 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790376259650642-11032-0/minicluster-data/master-0/wal/instance
I20260925 22:48:57.227716 11032 external_mini_cluster.cc:1428] Setting key 64a6e29431bc461066f7ed760190223c
I20260925 22:48:57.228200 32125 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:48:57.236294 32125 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 30f858a2009643eca6321f8b49c902ba: Bootstrap starting.
I20260925 22:48:57.238957 32125 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 30f858a2009643eca6321f8b49c902ba: Neither blocks nor log segments found. Creating new log.
I20260925 22:48:57.239282 32128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:48:57.229634 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.97:38401 (local address 127.0.0.1:34453)
0925 22:48:57.230187 (+ 553us) server_negotiation.cc:206] Beginning negotiation
0925 22:48:57.230210 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0925 22:48:57.230252 (+ 42us) server_negotiation.cc:407] Connection header received
0925 22:48:57.231227 (+ 975us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:48:57.231252 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:48:57.231705 (+ 453us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:48:57.232240 (+ 535us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:48:57.232765 (+ 525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:48:57.233910 (+ 1145us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:48:57.235050 (+ 1140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:48:57.235667 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:48:57.235941 (+ 274us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:48:57.237129 (+ 1188us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:48:57.237135 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:48:57.237151 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 22:48:57.237630 (+ 479us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:48:57.237701 (+ 71us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 22:48:57.237729 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:48:57.237910 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0925 22:48:57.238141 (+ 231us) server_negotiation.cc:299] Negotiation successful
0925 22:48:57.238319 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":384,"thread_start_us":181,"threads_started":1}
I20260925 22:48:57.239944 32125 log.cc:824] T 00000000000000000000000000000000 P 30f858a2009643eca6321f8b49c902ba: Log is configured to *not* fsync() on all Append() calls
I20260925 22:48:57.242812 32125 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 30f858a2009643eca6321f8b49c902ba: No bootstrap required, opened a new log
I20260925 22:48:57.244168 11032 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 22:48:57.247248 32125 raft_consensus.cc:348] T 00000000000000000000000000000000 P 30f858a2009643eca6321f8b49c902ba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30f858a2009643eca6321f8b49c902ba" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34453 } }
I20260925 22:48:57.247658 32125 raft_consensus.cc:374] T 00000000000000000000000000000000 P 30f858a2009643eca6321f8b49c902ba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:48:57.247749 32125 raft_consensus.cc:729] T 00000000000000000000000000000000 P 30f858a2009643eca6321f8b49c902ba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30f858a2009643eca6321f8b49c902ba, State: Initialized, Role: FOLLOWER
I20260925 22:48:57.248309 32125 consensus_queue.cc:260] T 00000000000000000000000000000000 P 30f858a2009643eca6321f8b49c902ba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30f858a2009643eca6321f8b49c902ba" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34453 } }
I20260925 22:48:57.248458 32125 raft_consensus.cc:388] T 00000000000000000000000000000000 P 30f858a2009643eca6321f8b49c902ba [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 22:48:57.248557 32125 raft_consensus.cc:482] T 00000000000000000000000000000000 P 30f858a2009643eca6321f8b49c902ba [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 22:48:57.248692 32125 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 30f858a2009643eca6321f8b49c902ba [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:48:57.249836 32125 raft_consensus.cc:504] T 00000000000000000000000000000000 P 30f858a2009643eca6321f8b49c902ba [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30f858a2009643eca6321f8b49c902ba" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34453 } }
I20260925 22:48:57.250275 32125 leader_election.cc:302] T 00000000000000000000000000000000 P 30f858a2009643eca6321f8b49c902ba [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 30f858a2009643eca6321f8b49c902ba; no voters:
I20260925 22:48:57.250782 32125 leader_election.cc:288] T 00000000000000000000000000000000 P 30f858a2009643eca6321f8b49c902ba [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 22:48:57.250864 32135 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 30f858a2009643eca6321f8b49c902ba [term 1 FOLLOWER]: Leader election won for term 1
I20260925 22:48:57.251072 32135 raft_consensus.cc:686] T 00000000000000000000000000000000 P 30f858a2009643eca6321f8b49c902ba [term 1 LEADER]: Becoming Leader. State: Replica: 30f858a2009643eca6321f8b49c902ba, State: Running, Role: LEADER
I20260925 22:48:57.251636 32128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:48:57.246901 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.97:42525 (local address 127.0.0.1:34453)
0925 22:48:57.247067 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0925 22:48:57.247072 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:48:57.247260 (+ 188us) server_negotiation.cc:407] Connection header received
0925 22:48:57.247407 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:48:57.247410 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:48:57.247461 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:48:57.247540 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:48:57.247968 (+ 428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:48:57.248655 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:48:57.249554 (+ 899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:48:57.249837 (+ 283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:48:57.249973 (+ 136us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:48:57.250935 (+ 962us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:48:57.250941 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:48:57.250946 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 22:48:57.250965 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:48:57.251001 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 22:48:57.251015 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:48:57.251147 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0925 22:48:57.251389 (+ 242us) server_negotiation.cc:299] Negotiation successful
0925 22:48:57.251433 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260925 22:48:57.251469 32135 consensus_queue.cc:237] T 00000000000000000000000000000000 P 30f858a2009643eca6321f8b49c902ba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30f858a2009643eca6321f8b49c902ba" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34453 } }
I20260925 22:48:57.252403 32125 sys_catalog.cc:564] T 00000000000000000000000000000000 P 30f858a2009643eca6321f8b49c902ba [sys.catalog]: configured and running, proceeding with master startup.
I20260925 22:48:57.254166 32136 sys_catalog.cc:455] T 00000000000000000000000000000000 P 30f858a2009643eca6321f8b49c902ba [sys.catalog]: SysCatalogTable state changed. Reason: New leader 30f858a2009643eca6321f8b49c902ba. Latest consensus state: current_term: 1 leader_uuid: "30f858a2009643eca6321f8b49c902ba" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30f858a2009643eca6321f8b49c902ba" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34453 } } }
I20260925 22:48:57.254256 32137 sys_catalog.cc:455] T 00000000000000000000000000000000 P 30f858a2009643eca6321f8b49c902ba [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "30f858a2009643eca6321f8b49c902ba" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30f858a2009643eca6321f8b49c902ba" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34453 } } }
I20260925 22:48:57.254357 32136 sys_catalog.cc:458] T 00000000000000000000000000000000 P 30f858a2009643eca6321f8b49c902ba [sys.catalog]: This master's current role is: LEADER
I20260925 22:48:57.254387 32137 sys_catalog.cc:458] T 00000000000000000000000000000000 P 30f858a2009643eca6321f8b49c902ba [sys.catalog]: This master's current role is: LEADER
I20260925 22:48:57.254791 32144 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 22:48:57.258622 32144 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 22:48:57.264791 32144 catalog_manager.cc:1347] Generated new cluster ID: 3ede46ec80034c6fad4b37dd4dd28cb2
I20260925 22:48:57.264886 32144 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 22:48:57.280159 32144 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 22:48:57.281860 32144 catalog_manager.cc:1504] Loading token signing keys...
I20260925 22:48:57.289891 32144 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 30f858a2009643eca6321f8b49c902ba: Generated new TSK 0
I20260925 22:48:57.290892 32144 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 22:48:57.305626 11032 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrWXm8X/build/debug/bin/kudu with pid 32057
W20260925 22:48:57.311224 32042 connection.cc:535] client connection to 127.0.0.1:34453 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:34453): Connection reset by peer (error 104)
2026-09-25T22:48:57Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1482 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790376259650642-11032-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 22:48:57 dist-test-slave-5vdr krb5kdc[32156](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 22:48:57 dist-test-slave-5vdr krb5kdc[32156](info): set up 2 sockets
Sep 25 22:48:57 dist-test-slave-5vdr krb5kdc[32156](info): commencing operation
krb5kdc: starting...
W20260925 22:48:58.392310 11032 mini_kdc.cc:121] Time spent starting KDC: real 1.066s user 0.007s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790376259650642-11032-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790376259650642-11032-0/krb5kdc/test-user.keytab.
Sep 25 22:48:58 dist-test-slave-5vdr krb5kdc[32156](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376538, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T22:48:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T22:48:58Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790376259650642-11032-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790376259650642-11032-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 22:48:58.522760 11032 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrWXm8X/build/debug/bin/kudu
/tmp/dist-test-taskrWXm8X/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790376259650642-11032-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790376259650642-11032-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790376259650642-11032-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790376259650642-11032-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44595
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790376259650642-11032-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:46551
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44595
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790376259650642-11032-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790376259650642-11032-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790376259650642-11032-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:48:58.643786 32169 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:48:58.644155 32169 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:48:58.644201 32169 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:48:58.648685 32169 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 22:48:58.648778 32169 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:48:58.648798 32169 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:48:58.648816 32169 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 22:48:58.648833 32169 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 22:48:58.655170 32169 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:46551
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790376259650642-11032-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790376259650642-11032-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44595
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790376259650642-11032-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44595
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790376259650642-11032-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32169
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790376259650642-11032-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:09:16 UTC on ccycloud.quasar-afmhyw.root.comops.site
build id 4673379
I20260925 22:48:58.656550 32169 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:48:58.657711 32169 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:48:58.667303 32175 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:48:58.667303 32174 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:48:58.667394 32177 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:48:58.667632 32169 server_base.cc:1034] running on GCE node
I20260925 22:48:58.668357 32169 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:48:58.669677 32169 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:48:58.670908 32169 hybrid_clock.cc:648] HybridClock initialized: now 1790376538670885 us; error 53 us; skew 500 ppm
Sep 25 22:48:58 dist-test-slave-5vdr krb5kdc[32156](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376538, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:48:58.674327 32169 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 22:48:58.675489 32169 webserver.cc:460] Webserver started at http://127.0.0.1:45723/ using document root <none> and password file <none>
I20260925 22:48:58.676223 32169 fs_manager.cc:359] Metadata directory not provided
I20260925 22:48:58.676337 32169 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:48:58.676605 32169 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:48:58.679013 32169 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790376259650642-11032-0/minicluster-data/master-0/data/instance:
uuid: "f4277e603b9a4711b76a34f95d064457"
format_stamp: "Formatted at 2026-09-25 22:48:58 on dist-test-slave-5vdr"
server_key: "00d9f1fa7025d9dd406d3edf8ee3de4f"
server_key_iv: "d6ff0921a873884008dcabeda1d563e7"
server_key_version: "encryptionkey@0"
I20260925 22:48:58.679692 32169 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790376259650642-11032-0/minicluster-data/master-0/wal/instance:
uuid: "f4277e603b9a4711b76a34f95d064457"
format_stamp: "Formatted at 2026-09-25 22:48:58 on dist-test-slave-5vdr"
server_key: "00d9f1fa7025d9dd406d3edf8ee3de4f"
server_key_iv: "d6ff0921a873884008dcabeda1d563e7"
server_key_version: "encryptionkey@0"
I20260925 22:48:58.684116 32169 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.003s
I20260925 22:48:58.687057 32184 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:48:58.688671 32169 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 22:48:58.688872 32169 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790376259650642-11032-0/minicluster-data/master-0/data,/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790376259650642-11032-0/minicluster-data/master-0/wal
uuid: "f4277e603b9a4711b76a34f95d064457"
format_stamp: "Formatted at 2026-09-25 22:48:58 on dist-test-slave-5vdr"
server_key: "00d9f1fa7025d9dd406d3edf8ee3de4f"
server_key_iv: "d6ff0921a873884008dcabeda1d563e7"
server_key_version: "encryptionkey@0"
I20260925 22:48:58.689024 32169 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790376259650642-11032-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790376259650642-11032-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790376259650642-11032-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:48:58.694132 32169 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:48:58.698014 32169 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:48:58.698262 32169 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:48:58.706570 32169 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44595
I20260925 22:48:58.706585 32236 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44595 every 8 connection(s)
I20260925 22:48:58.707700 32169 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790376259650642-11032-0/minicluster-data/master-0/data/info.pb
I20260925 22:48:58.710322 11032 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrWXm8X/build/debug/bin/kudu as pid 32169
I20260925 22:48:58.710490 11032 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790376259650642-11032-0/minicluster-data/master-0/wal/instance
I20260925 22:48:58.710815 11032 external_mini_cluster.cc:1428] Setting key 2af3dbd05a0ff3f76a4714f5a4c9f465
I20260925 22:48:58.711172 32237 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:48:58.716723 32240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:48:58.712470 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.97:36745 (local address 127.0.0.1:44595)
0925 22:48:58.712970 (+ 500us) server_negotiation.cc:206] Beginning negotiation
0925 22:48:58.712987 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0925 22:48:58.713025 (+ 38us) server_negotiation.cc:407] Connection header received
0925 22:48:58.713717 (+ 692us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:48:58.713735 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:48:58.714075 (+ 340us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:48:58.714457 (+ 382us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:48:58.715372 (+ 915us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:48:58.715376 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:48:58.715389 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 22:48:58.715402 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:48:58.715460 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 22:48:58.715481 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:48:58.715664 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0925 22:48:58.715740 (+ 76us) server_negotiation.cc:299] Negotiation successful
0925 22:48:58.715958 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":345,"thread_start_us":166,"threads_started":1}
I20260925 22:48:58.718523 32237 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f4277e603b9a4711b76a34f95d064457: Bootstrap starting.
I20260925 22:48:58.719897 11032 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 22:48:58.721627 32237 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f4277e603b9a4711b76a34f95d064457: Neither blocks nor log segments found. Creating new log.
I20260925 22:48:58.722833 32237 log.cc:824] T 00000000000000000000000000000000 P f4277e603b9a4711b76a34f95d064457: Log is configured to *not* fsync() on all Append() calls
I20260925 22:48:58.724783 32240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:48:58.722480 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.97:37417 (local address 127.0.0.1:44595)
0925 22:48:58.722721 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0925 22:48:58.722726 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:48:58.722913 (+ 187us) server_negotiation.cc:407] Connection header received
0925 22:48:58.723111 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:48:58.723114 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:48:58.723154 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:48:58.723257 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:48:58.724209 (+ 952us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:48:58.724213 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:48:58.724215 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 22:48:58.724228 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:48:58.724259 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 22:48:58.724272 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:48:58.724374 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0925 22:48:58.724603 (+ 229us) server_negotiation.cc:299] Negotiation successful
0925 22:48:58.724651 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":144}
I20260925 22:48:58.725704 32237 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f4277e603b9a4711b76a34f95d064457: No bootstrap required, opened a new log
I20260925 22:48:58.729782 32237 raft_consensus.cc:348] T 00000000000000000000000000000000 P f4277e603b9a4711b76a34f95d064457 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f4277e603b9a4711b76a34f95d064457" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44595 } }
I20260925 22:48:58.730123 32237 raft_consensus.cc:374] T 00000000000000000000000000000000 P f4277e603b9a4711b76a34f95d064457 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:48:58.730198 32237 raft_consensus.cc:729] T 00000000000000000000000000000000 P f4277e603b9a4711b76a34f95d064457 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f4277e603b9a4711b76a34f95d064457, State: Initialized, Role: FOLLOWER
I20260925 22:48:58.730705 32237 consensus_queue.cc:260] T 00000000000000000000000000000000 P f4277e603b9a4711b76a34f95d064457 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f4277e603b9a4711b76a34f95d064457" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44595 } }
I20260925 22:48:58.730887 32237 raft_consensus.cc:388] T 00000000000000000000000000000000 P f4277e603b9a4711b76a34f95d064457 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 22:48:58.730984 32237 raft_consensus.cc:482] T 00000000000000000000000000000000 P f4277e603b9a4711b76a34f95d064457 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 22:48:58.731115 32237 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f4277e603b9a4711b76a34f95d064457 [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:48:58.732372 32237 raft_consensus.cc:504] T 00000000000000000000000000000000 P f4277e603b9a4711b76a34f95d064457 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f4277e603b9a4711b76a34f95d064457" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44595 } }
I20260925 22:48:58.732803 32237 leader_election.cc:302] T 00000000000000000000000000000000 P f4277e603b9a4711b76a34f95d064457 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f4277e603b9a4711b76a34f95d064457; no voters:
I20260925 22:48:58.733202 32237 leader_election.cc:288] T 00000000000000000000000000000000 P f4277e603b9a4711b76a34f95d064457 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 22:48:58.733385 32247 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f4277e603b9a4711b76a34f95d064457 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 22:48:58.733666 32247 raft_consensus.cc:686] T 00000000000000000000000000000000 P f4277e603b9a4711b76a34f95d064457 [term 1 LEADER]: Becoming Leader. State: Replica: f4277e603b9a4711b76a34f95d064457, State: Running, Role: LEADER
I20260925 22:48:58.734027 32247 consensus_queue.cc:237] T 00000000000000000000000000000000 P f4277e603b9a4711b76a34f95d064457 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f4277e603b9a4711b76a34f95d064457" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44595 } }
I20260925 22:48:58.734814 32237 sys_catalog.cc:564] T 00000000000000000000000000000000 P f4277e603b9a4711b76a34f95d064457 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 22:48:58.735967 32249 sys_catalog.cc:455] T 00000000000000000000000000000000 P f4277e603b9a4711b76a34f95d064457 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f4277e603b9a4711b76a34f95d064457. Latest consensus state: current_term: 1 leader_uuid: "f4277e603b9a4711b76a34f95d064457" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f4277e603b9a4711b76a34f95d064457" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44595 } } }
I20260925 22:48:58.736145 32249 sys_catalog.cc:458] T 00000000000000000000000000000000 P f4277e603b9a4711b76a34f95d064457 [sys.catalog]: This master's current role is: LEADER
I20260925 22:48:58.736426 32248 sys_catalog.cc:455] T 00000000000000000000000000000000 P f4277e603b9a4711b76a34f95d064457 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f4277e603b9a4711b76a34f95d064457" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f4277e603b9a4711b76a34f95d064457" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44595 } } }
I20260925 22:48:58.736534 32248 sys_catalog.cc:458] T 00000000000000000000000000000000 P f4277e603b9a4711b76a34f95d064457 [sys.catalog]: This master's current role is: LEADER
I20260925 22:48:58.736671 32256 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 22:48:58.740355 32256 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 22:48:58.747896 32256 catalog_manager.cc:1347] Generated new cluster ID: 1b4340bbdabf4fb8beb6f9ebefe489fe
I20260925 22:48:58.748026 32256 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 22:48:58.769376 32256 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 22:48:58.769778 32256 catalog_manager.cc:1504] Loading token signing keys...
I20260925 22:48:58.775745 32256 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f4277e603b9a4711b76a34f95d064457: Generated new TSK 0
I20260925 22:48:58.776754 32256 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 22:48:58.784040 11032 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrWXm8X/build/debug/bin/kudu with pid 32169
2026-09-25T22:48:58Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1478 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790376259650642-11032-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 22:48:58 dist-test-slave-5vdr krb5kdc[32268](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 22:48:58 dist-test-slave-5vdr krb5kdc[32268](info): set up 2 sockets
Sep 25 22:48:58 dist-test-slave-5vdr krb5kdc[32268](info): commencing operation
krb5kdc: starting...
W20260925 22:48:59.836310 11032 mini_kdc.cc:121] Time spent starting KDC: real 1.031s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790376259650642-11032-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790376259650642-11032-0/krb5kdc/test-user.keytab.
Sep 25 22:48:59 dist-test-slave-5vdr krb5kdc[32268](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376539, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T22:48:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T22:48:59Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790376259650642-11032-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790376259650642-11032-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 22:48:59.967777 11032 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrWXm8X/build/debug/bin/kudu
/tmp/dist-test-taskrWXm8X/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790376259650642-11032-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790376259650642-11032-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790376259650642-11032-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790376259650642-11032-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46815
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790376259650642-11032-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38193
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:46815
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790376259650642-11032-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790376259650642-11032-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790376259650642-11032-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:49:00.090241 32281 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:49:00.090636 32281 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:49:00.090725 32281 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:49:00.095408 32281 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 22:49:00.095571 32281 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:49:00.095614 32281 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:49:00.095655 32281 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 22:49:00.095688 32281 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 22:49:00.101747 32281 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38193
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790376259650642-11032-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790376259650642-11032-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:46815
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790376259650642-11032-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46815
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790376259650642-11032-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32281
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790376259650642-11032-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:09:16 UTC on ccycloud.quasar-afmhyw.root.comops.site
build id 4673379
I20260925 22:49:00.103009 32281 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:49:00.104187 32281 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:49:00.113915 32289 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:49:00.113936 32287 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:49:00.113920 32286 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:49:00.114135 32281 server_base.cc:1034] running on GCE node
I20260925 22:49:00.114945 32281 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:49:00.116279 32281 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:49:00.117497 32281 hybrid_clock.cc:648] HybridClock initialized: now 1790376540117482 us; error 35 us; skew 500 ppm
Sep 25 22:49:00 dist-test-slave-5vdr krb5kdc[32268](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376540, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:49:00.120842 32281 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 22:49:00.122285 32281 webserver.cc:460] Webserver started at http://127.0.0.1:38689/ using document root <none> and password file <none>
I20260925 22:49:00.122997 32281 fs_manager.cc:359] Metadata directory not provided
I20260925 22:49:00.123112 32281 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:49:00.123373 32281 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:49:00.125860 32281 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790376259650642-11032-0/minicluster-data/master-0/data/instance:
uuid: "30f58e59ba764c5fa7d6154ba1c3764f"
format_stamp: "Formatted at 2026-09-25 22:49:00 on dist-test-slave-5vdr"
server_key: "bb467ace2daaf464ceae6e08fef08e48"
server_key_iv: "40d70a1a810587e9d0f520c26e29b286"
server_key_version: "encryptionkey@0"
I20260925 22:49:00.126549 32281 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790376259650642-11032-0/minicluster-data/master-0/wal/instance:
uuid: "30f58e59ba764c5fa7d6154ba1c3764f"
format_stamp: "Formatted at 2026-09-25 22:49:00 on dist-test-slave-5vdr"
server_key: "bb467ace2daaf464ceae6e08fef08e48"
server_key_iv: "40d70a1a810587e9d0f520c26e29b286"
server_key_version: "encryptionkey@0"
I20260925 22:49:00.131018 32281 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.001s
I20260925 22:49:00.133957 32296 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:49:00.135399 32281 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260925 22:49:00.135587 32281 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790376259650642-11032-0/minicluster-data/master-0/data,/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790376259650642-11032-0/minicluster-data/master-0/wal
uuid: "30f58e59ba764c5fa7d6154ba1c3764f"
format_stamp: "Formatted at 2026-09-25 22:49:00 on dist-test-slave-5vdr"
server_key: "bb467ace2daaf464ceae6e08fef08e48"
server_key_iv: "40d70a1a810587e9d0f520c26e29b286"
server_key_version: "encryptionkey@0"
I20260925 22:49:00.135732 32281 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790376259650642-11032-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790376259650642-11032-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790376259650642-11032-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:49:00.156131 32281 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:49:00.159605 32281 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:49:00.159917 32281 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:49:00.168128 32281 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:46815
I20260925 22:49:00.168156 32348 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:46815 every 8 connection(s)
I20260925 22:49:00.169301 32281 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790376259650642-11032-0/minicluster-data/master-0/data/info.pb
I20260925 22:49:00.172547 32349 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:49:00.175515 11032 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrWXm8X/build/debug/bin/kudu as pid 32281
I20260925 22:49:00.175688 11032 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790376259650642-11032-0/minicluster-data/master-0/wal/instance
I20260925 22:49:00.176074 11032 external_mini_cluster.cc:1428] Setting key 916c50e40780de4ee4844422d4daa462
I20260925 22:49:00.180203 32349 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 30f58e59ba764c5fa7d6154ba1c3764f: Bootstrap starting.
I20260925 22:49:00.183038 32349 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 30f58e59ba764c5fa7d6154ba1c3764f: Neither blocks nor log segments found. Creating new log.
I20260925 22:49:00.184041 32349 log.cc:824] T 00000000000000000000000000000000 P 30f58e59ba764c5fa7d6154ba1c3764f: Log is configured to *not* fsync() on all Append() calls
Sep 25 22:49:00 dist-test-slave-5vdr krb5kdc[32268](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376539, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260925 22:49:00.186792 32349 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 30f58e59ba764c5fa7d6154ba1c3764f: No bootstrap required, opened a new log
I20260925 22:49:00.190290 32349 raft_consensus.cc:348] T 00000000000000000000000000000000 P 30f58e59ba764c5fa7d6154ba1c3764f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30f58e59ba764c5fa7d6154ba1c3764f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46815 } }
I20260925 22:49:00.190618 32349 raft_consensus.cc:374] T 00000000000000000000000000000000 P 30f58e59ba764c5fa7d6154ba1c3764f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:49:00.190703 32349 raft_consensus.cc:729] T 00000000000000000000000000000000 P 30f58e59ba764c5fa7d6154ba1c3764f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30f58e59ba764c5fa7d6154ba1c3764f, State: Initialized, Role: FOLLOWER
I20260925 22:49:00.191283 32349 consensus_queue.cc:260] T 00000000000000000000000000000000 P 30f58e59ba764c5fa7d6154ba1c3764f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30f58e59ba764c5fa7d6154ba1c3764f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46815 } }
I20260925 22:49:00.191473 32349 raft_consensus.cc:388] T 00000000000000000000000000000000 P 30f58e59ba764c5fa7d6154ba1c3764f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 22:49:00.191591 32349 raft_consensus.cc:482] T 00000000000000000000000000000000 P 30f58e59ba764c5fa7d6154ba1c3764f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 22:49:00.191720 32349 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 30f58e59ba764c5fa7d6154ba1c3764f [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:49:00.192965 32349 raft_consensus.cc:504] T 00000000000000000000000000000000 P 30f58e59ba764c5fa7d6154ba1c3764f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30f58e59ba764c5fa7d6154ba1c3764f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46815 } }
I20260925 22:49:00.193269 32352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:00.177834 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.97:41785 (local address 127.0.0.1:46815)
0925 22:49:00.178401 (+ 567us) server_negotiation.cc:206] Beginning negotiation
0925 22:49:00.178417 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0925 22:49:00.178456 (+ 39us) server_negotiation.cc:407] Connection header received
0925 22:49:00.179152 (+ 696us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:49:00.179172 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:49:00.179573 (+ 401us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:49:00.180000 (+ 427us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:49:00.180938 (+ 938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:49:00.182147 (+ 1209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:49:00.183128 (+ 981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:49:00.183767 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:49:00.183999 (+ 232us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:49:00.187258 (+ 3259us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:49:00.187265 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:49:00.187282 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:49:00.187317 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:49:00.189745 (+ 2428us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:49:00.190412 (+ 667us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:49:00.190418 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:49:00.190424 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:49:00.190553 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:49:00.190954 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:49:00.190962 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:49:00.190964 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:49:00.191410 (+ 446us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:49:00.191717 (+ 307us) server_negotiation.cc:1035] Waiting for connection context
0925 22:49:00.192100 (+ 383us) server_negotiation.cc:299] Negotiation successful
0925 22:49:00.192378 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":387,"thread_start_us":173,"threads_started":1}
I20260925 22:49:00.193401 32349 leader_election.cc:302] T 00000000000000000000000000000000 P 30f58e59ba764c5fa7d6154ba1c3764f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 30f58e59ba764c5fa7d6154ba1c3764f; no voters:
I20260925 22:49:00.193878 32349 leader_election.cc:288] T 00000000000000000000000000000000 P 30f58e59ba764c5fa7d6154ba1c3764f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 22:49:00.193980 32354 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 30f58e59ba764c5fa7d6154ba1c3764f [term 1 FOLLOWER]: Leader election won for term 1
I20260925 22:49:00.194187 32354 raft_consensus.cc:686] T 00000000000000000000000000000000 P 30f58e59ba764c5fa7d6154ba1c3764f [term 1 LEADER]: Becoming Leader. State: Replica: 30f58e59ba764c5fa7d6154ba1c3764f, State: Running, Role: LEADER
I20260925 22:49:00.195384 32349 sys_catalog.cc:564] T 00000000000000000000000000000000 P 30f58e59ba764c5fa7d6154ba1c3764f [sys.catalog]: configured and running, proceeding with master startup.
I20260925 22:49:00.195566 32354 consensus_queue.cc:237] T 00000000000000000000000000000000 P 30f58e59ba764c5fa7d6154ba1c3764f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30f58e59ba764c5fa7d6154ba1c3764f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46815 } }
I20260925 22:49:00.197232 11032 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 22:49:00.198315 32355 sys_catalog.cc:455] T 00000000000000000000000000000000 P 30f58e59ba764c5fa7d6154ba1c3764f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 30f58e59ba764c5fa7d6154ba1c3764f. Latest consensus state: current_term: 1 leader_uuid: "30f58e59ba764c5fa7d6154ba1c3764f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30f58e59ba764c5fa7d6154ba1c3764f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46815 } } }
I20260925 22:49:00.198092 32356 sys_catalog.cc:455] T 00000000000000000000000000000000 P 30f58e59ba764c5fa7d6154ba1c3764f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "30f58e59ba764c5fa7d6154ba1c3764f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30f58e59ba764c5fa7d6154ba1c3764f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46815 } } }
I20260925 22:49:00.198426 32355 sys_catalog.cc:458] T 00000000000000000000000000000000 P 30f58e59ba764c5fa7d6154ba1c3764f [sys.catalog]: This master's current role is: LEADER
I20260925 22:49:00.198426 32356 sys_catalog.cc:458] T 00000000000000000000000000000000 P 30f58e59ba764c5fa7d6154ba1c3764f [sys.catalog]: This master's current role is: LEADER
W20260925 22:49:00.202430 32368 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 30f58e59ba764c5fa7d6154ba1c3764f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 22:49:00.202576 32368 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 22:49:00.202697 32369 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 22:49:00.203662 32369 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 22:49:00.210345 32352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:00.199739 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.97:46095 (local address 127.0.0.1:46815)
0925 22:49:00.199913 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0925 22:49:00.199919 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 22:49:00.200345 (+ 426us) server_negotiation.cc:407] Connection header received
0925 22:49:00.200441 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:49:00.200444 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:49:00.200503 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:49:00.200618 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:49:00.201491 (+ 873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:49:00.202346 (+ 855us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:49:00.203630 (+ 1284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:49:00.204047 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:49:00.204180 (+ 133us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:49:00.206140 (+ 1960us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:49:00.206145 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:49:00.206148 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:49:00.206182 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:49:00.208039 (+ 1857us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:49:00.208747 (+ 708us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:49:00.208751 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:49:00.208753 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:49:00.208878 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:49:00.209285 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:49:00.209291 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:49:00.209293 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:49:00.209550 (+ 257us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:49:00.209717 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0925 22:49:00.210056 (+ 339us) server_negotiation.cc:299] Negotiation successful
0925 22:49:00.210177 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260925 22:49:00.211195 32369 catalog_manager.cc:1347] Generated new cluster ID: 09ecd95fb31b4e73b78f13c253fbd3f7
I20260925 22:49:00.211269 32369 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 22:49:00.230623 32369 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 22:49:00.232096 32369 catalog_manager.cc:1504] Loading token signing keys...
I20260925 22:49:00.250733 32369 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 30f58e59ba764c5fa7d6154ba1c3764f: Generated new TSK 0
I20260925 22:49:00.251732 32369 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 22:49:00.267464 11032 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrWXm8X/build/debug/bin/kudu with pid 32281
W20260925 22:49:00.273096 32266 connection.cc:535] client connection to 127.0.0.1:46815 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:46815): Connection reset by peer (error 104)
2026-09-25T22:49:00Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (1483 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790376259650642-11032-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 22:49:00 dist-test-slave-5vdr krb5kdc[32380](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 22:49:00 dist-test-slave-5vdr krb5kdc[32380](info): set up 2 sockets
Sep 25 22:49:00 dist-test-slave-5vdr krb5kdc[32380](info): commencing operation
krb5kdc: starting...
W20260925 22:49:01.316396 11032 mini_kdc.cc:121] Time spent starting KDC: real 1.028s user 0.006s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790376259650642-11032-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790376259650642-11032-0/krb5kdc/test-user.keytab.
Sep 25 22:49:01 dist-test-slave-5vdr krb5kdc[32380](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376541, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T22:49:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T22:49:01Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790376259650642-11032-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790376259650642-11032-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 22:49:01.445199 11032 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrWXm8X/build/debug/bin/kudu
/tmp/dist-test-taskrWXm8X/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790376259650642-11032-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790376259650642-11032-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790376259650642-11032-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790376259650642-11032-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45927
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790376259650642-11032-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:46009
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45927
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790376259650642-11032-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790376259650642-11032-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790376259650642-11032-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:49:01.564010 32393 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:49:01.564327 32393 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:49:01.564370 32393 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:49:01.568737 32393 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 22:49:01.568831 32393 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:49:01.568851 32393 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:49:01.568868 32393 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 22:49:01.568884 32393 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 22:49:01.574891 32393 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:46009
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790376259650642-11032-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790376259650642-11032-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45927
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790376259650642-11032-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45927
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790376259650642-11032-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32393
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790376259650642-11032-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:09:16 UTC on ccycloud.quasar-afmhyw.root.comops.site
build id 4673379
I20260925 22:49:01.576150 32393 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:49:01.577234 32393 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:49:01.586648 32401 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:49:01.586684 32398 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:49:01.586833 32393 server_base.cc:1034] running on GCE node
W20260925 22:49:01.586684 32399 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:49:01.587613 32393 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:49:01.588917 32393 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:49:01.590145 32393 hybrid_clock.cc:648] HybridClock initialized: now 1790376541590135 us; error 36 us; skew 500 ppm
Sep 25 22:49:01 dist-test-slave-5vdr krb5kdc[32380](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376541, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:49:01.593366 32393 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 22:49:01.594416 32393 webserver.cc:460] Webserver started at http://127.0.0.1:39193/ using document root <none> and password file <none>
I20260925 22:49:01.595006 32393 fs_manager.cc:359] Metadata directory not provided
I20260925 22:49:01.595078 32393 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:49:01.595259 32393 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:49:01.597600 32393 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790376259650642-11032-0/minicluster-data/master-0/data/instance:
uuid: "ccde520a124d4410a29543dc315f8f6b"
format_stamp: "Formatted at 2026-09-25 22:49:01 on dist-test-slave-5vdr"
server_key: "bcf008bf6f70e104e82653f4aa2e0c6b"
server_key_iv: "492fc6bd3014073ae569840c277d9126"
server_key_version: "encryptionkey@0"
I20260925 22:49:01.598193 32393 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790376259650642-11032-0/minicluster-data/master-0/wal/instance:
uuid: "ccde520a124d4410a29543dc315f8f6b"
format_stamp: "Formatted at 2026-09-25 22:49:01 on dist-test-slave-5vdr"
server_key: "bcf008bf6f70e104e82653f4aa2e0c6b"
server_key_iv: "492fc6bd3014073ae569840c277d9126"
server_key_version: "encryptionkey@0"
I20260925 22:49:01.602461 32393 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260925 22:49:01.605410 32408 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:49:01.606886 32393 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260925 22:49:01.607044 32393 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790376259650642-11032-0/minicluster-data/master-0/data,/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790376259650642-11032-0/minicluster-data/master-0/wal
uuid: "ccde520a124d4410a29543dc315f8f6b"
format_stamp: "Formatted at 2026-09-25 22:49:01 on dist-test-slave-5vdr"
server_key: "bcf008bf6f70e104e82653f4aa2e0c6b"
server_key_iv: "492fc6bd3014073ae569840c277d9126"
server_key_version: "encryptionkey@0"
I20260925 22:49:01.607158 32393 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790376259650642-11032-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790376259650642-11032-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790376259650642-11032-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:49:01.656013 32393 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:49:01.659771 32393 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:49:01.660058 32393 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:49:01.668498 32393 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45927
I20260925 22:49:01.668525 32460 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45927 every 8 connection(s)
I20260925 22:49:01.669636 32393 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790376259650642-11032-0/minicluster-data/master-0/data/info.pb
I20260925 22:49:01.672976 32461 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:49:01.673287 11032 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrWXm8X/build/debug/bin/kudu as pid 32393
I20260925 22:49:01.673451 11032 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790376259650642-11032-0/minicluster-data/master-0/wal/instance
I20260925 22:49:01.673769 11032 external_mini_cluster.cc:1428] Setting key 96da2295455acb2ec20c79de80042641
I20260925 22:49:01.681618 32461 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ccde520a124d4410a29543dc315f8f6b: Bootstrap starting.
I20260925 22:49:01.684711 32461 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ccde520a124d4410a29543dc315f8f6b: Neither blocks nor log segments found. Creating new log.
I20260925 22:49:01.685624 32464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:01.675381 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.97:56819 (local address 127.0.0.1:45927)
0925 22:49:01.675943 (+ 562us) server_negotiation.cc:206] Beginning negotiation
0925 22:49:01.675967 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0925 22:49:01.676018 (+ 51us) server_negotiation.cc:407] Connection header received
0925 22:49:01.676854 (+ 836us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:49:01.676884 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:49:01.677470 (+ 586us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:49:01.677909 (+ 439us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:49:01.678372 (+ 463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:49:01.680086 (+ 1714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:49:01.681165 (+ 1079us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:49:01.681920 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:49:01.682113 (+ 193us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:49:01.683222 (+ 1109us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:49:01.683230 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:49:01.683247 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 22:49:01.683819 (+ 572us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:49:01.683980 (+ 161us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 22:49:01.684019 (+ 39us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:49:01.684176 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0925 22:49:01.684400 (+ 224us) server_negotiation.cc:299] Negotiation successful
0925 22:49:01.684563 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":347,"thread_start_us":146,"threads_started":1}
I20260925 22:49:01.685714 32461 log.cc:824] T 00000000000000000000000000000000 P ccde520a124d4410a29543dc315f8f6b: Log is configured to *not* fsync() on all Append() calls
I20260925 22:49:01.688266 32461 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ccde520a124d4410a29543dc315f8f6b: No bootstrap required, opened a new log
I20260925 22:49:01.689110 11032 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 22:49:01.693015 32461 raft_consensus.cc:348] T 00000000000000000000000000000000 P ccde520a124d4410a29543dc315f8f6b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ccde520a124d4410a29543dc315f8f6b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45927 } }
I20260925 22:49:01.693411 32461 raft_consensus.cc:374] T 00000000000000000000000000000000 P ccde520a124d4410a29543dc315f8f6b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:49:01.693487 32461 raft_consensus.cc:729] T 00000000000000000000000000000000 P ccde520a124d4410a29543dc315f8f6b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ccde520a124d4410a29543dc315f8f6b, State: Initialized, Role: FOLLOWER
I20260925 22:49:01.694031 32461 consensus_queue.cc:260] T 00000000000000000000000000000000 P ccde520a124d4410a29543dc315f8f6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ccde520a124d4410a29543dc315f8f6b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45927 } }
I20260925 22:49:01.694255 32461 raft_consensus.cc:388] T 00000000000000000000000000000000 P ccde520a124d4410a29543dc315f8f6b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 22:49:01.694360 32461 raft_consensus.cc:482] T 00000000000000000000000000000000 P ccde520a124d4410a29543dc315f8f6b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 22:49:01.694504 32461 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ccde520a124d4410a29543dc315f8f6b [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:49:01.696025 32461 raft_consensus.cc:504] T 00000000000000000000000000000000 P ccde520a124d4410a29543dc315f8f6b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ccde520a124d4410a29543dc315f8f6b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45927 } }
I20260925 22:49:01.696288 32464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:01.691324 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.97:38729 (local address 127.0.0.1:45927)
0925 22:49:01.691460 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0925 22:49:01.691464 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:49:01.691753 (+ 289us) server_negotiation.cc:407] Connection header received
0925 22:49:01.691931 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:49:01.691933 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:49:01.691983 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:49:01.692061 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:49:01.692453 (+ 392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:49:01.693016 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:49:01.694173 (+ 1157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:49:01.694600 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:49:01.694712 (+ 112us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:49:01.695674 (+ 962us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:49:01.695678 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:49:01.695680 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 22:49:01.695695 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:49:01.695723 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 22:49:01.695735 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:49:01.695805 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0925 22:49:01.696068 (+ 263us) server_negotiation.cc:299] Negotiation successful
0925 22:49:01.696105 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260925 22:49:01.696506 32461 leader_election.cc:302] T 00000000000000000000000000000000 P ccde520a124d4410a29543dc315f8f6b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ccde520a124d4410a29543dc315f8f6b; no voters:
I20260925 22:49:01.696985 32461 leader_election.cc:288] T 00000000000000000000000000000000 P ccde520a124d4410a29543dc315f8f6b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 22:49:01.697117 32471 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ccde520a124d4410a29543dc315f8f6b [term 1 FOLLOWER]: Leader election won for term 1
I20260925 22:49:01.697535 32471 raft_consensus.cc:686] T 00000000000000000000000000000000 P ccde520a124d4410a29543dc315f8f6b [term 1 LEADER]: Becoming Leader. State: Replica: ccde520a124d4410a29543dc315f8f6b, State: Running, Role: LEADER
I20260925 22:49:01.697893 32471 consensus_queue.cc:237] T 00000000000000000000000000000000 P ccde520a124d4410a29543dc315f8f6b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ccde520a124d4410a29543dc315f8f6b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45927 } }
I20260925 22:49:01.698432 32461 sys_catalog.cc:564] T 00000000000000000000000000000000 P ccde520a124d4410a29543dc315f8f6b [sys.catalog]: configured and running, proceeding with master startup.
I20260925 22:49:01.699100 32471 sys_catalog.cc:455] T 00000000000000000000000000000000 P ccde520a124d4410a29543dc315f8f6b [sys.catalog]: SysCatalogTable state changed. Reason: New leader ccde520a124d4410a29543dc315f8f6b. Latest consensus state: current_term: 1 leader_uuid: "ccde520a124d4410a29543dc315f8f6b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ccde520a124d4410a29543dc315f8f6b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45927 } } }
I20260925 22:49:01.699244 32471 sys_catalog.cc:458] T 00000000000000000000000000000000 P ccde520a124d4410a29543dc315f8f6b [sys.catalog]: This master's current role is: LEADER
I20260925 22:49:01.699693 32473 sys_catalog.cc:455] T 00000000000000000000000000000000 P ccde520a124d4410a29543dc315f8f6b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ccde520a124d4410a29543dc315f8f6b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ccde520a124d4410a29543dc315f8f6b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45927 } } }
I20260925 22:49:01.699880 32473 sys_catalog.cc:458] T 00000000000000000000000000000000 P ccde520a124d4410a29543dc315f8f6b [sys.catalog]: This master's current role is: LEADER
I20260925 22:49:01.700150 32476 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 22:49:01.703949 32476 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 22:49:01.710826 32476 catalog_manager.cc:1347] Generated new cluster ID: 25711fac464f4803bbf068ec4438fbb3
I20260925 22:49:01.710930 32476 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 22:49:01.722568 32476 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 22:49:01.723752 32476 catalog_manager.cc:1504] Loading token signing keys...
I20260925 22:49:01.731873 32476 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ccde520a124d4410a29543dc315f8f6b: Generated new TSK 0
I20260925 22:49:01.732681 32476 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 22:49:01.742553 11032 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrWXm8X/build/debug/bin/kudu with pid 32393
W20260925 22:49:01.747574 32378 connection.cc:535] client connection to 127.0.0.1:45927 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:45927): Connection reset by peer (error 104)
2026-09-25T22:49:01Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1474 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790376259650642-11032-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 22:49:01 dist-test-slave-5vdr krb5kdc[32492](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 22:49:01 dist-test-slave-5vdr krb5kdc[32492](info): set up 2 sockets
Sep 25 22:49:01 dist-test-slave-5vdr krb5kdc[32492](info): commencing operation
krb5kdc: starting...
W20260925 22:49:02.792189 11032 mini_kdc.cc:121] Time spent starting KDC: real 1.030s user 0.004s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790376259650642-11032-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790376259650642-11032-0/krb5kdc/test-user.keytab.
Sep 25 22:49:02 dist-test-slave-5vdr krb5kdc[32492](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376542, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T22:49:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T22:49:02Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790376259650642-11032-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790376259650642-11032-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 22:49:02.923415 11032 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrWXm8X/build/debug/bin/kudu
/tmp/dist-test-taskrWXm8X/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790376259650642-11032-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790376259650642-11032-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790376259650642-11032-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790376259650642-11032-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40251
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790376259650642-11032-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44597
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40251
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790376259650642-11032-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790376259650642-11032-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790376259650642-11032-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:49:03.044827 32505 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:49:03.045204 32505 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:49:03.045285 32505 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:49:03.049726 32505 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 22:49:03.049844 32505 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:49:03.049885 32505 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:49:03.049925 32505 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 22:49:03.049965 32505 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 22:49:03.055914 32505 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44597
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790376259650642-11032-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790376259650642-11032-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40251
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790376259650642-11032-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40251
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790376259650642-11032-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32505
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790376259650642-11032-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:09:16 UTC on ccycloud.quasar-afmhyw.root.comops.site
build id 4673379
I20260925 22:49:03.057171 32505 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:49:03.058306 32505 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:49:03.068162 32510 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:49:03.068277 32513 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:49:03.068204 32505 server_base.cc:1034] running on GCE node
W20260925 22:49:03.068212 32511 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:49:03.069068 32505 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:49:03.070377 32505 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:49:03.071584 32505 hybrid_clock.cc:648] HybridClock initialized: now 1790376543071547 us; error 57 us; skew 500 ppm
Sep 25 22:49:03 dist-test-slave-5vdr krb5kdc[32492](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376543, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:49:03.075080 32505 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 22:49:03.076264 32505 webserver.cc:460] Webserver started at http://127.0.0.1:34325/ using document root <none> and password file <none>
I20260925 22:49:03.076931 32505 fs_manager.cc:359] Metadata directory not provided
I20260925 22:49:03.077006 32505 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:49:03.077260 32505 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:49:03.079579 32505 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790376259650642-11032-0/minicluster-data/master-0/data/instance:
uuid: "51b835858413449380be32af2f930a6b"
format_stamp: "Formatted at 2026-09-25 22:49:03 on dist-test-slave-5vdr"
server_key: "689d7e0ea64315aad666571777886099"
server_key_iv: "aa590f21f11bd75bd0f8d97a74a83082"
server_key_version: "encryptionkey@0"
I20260925 22:49:03.080314 32505 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790376259650642-11032-0/minicluster-data/master-0/wal/instance:
uuid: "51b835858413449380be32af2f930a6b"
format_stamp: "Formatted at 2026-09-25 22:49:03 on dist-test-slave-5vdr"
server_key: "689d7e0ea64315aad666571777886099"
server_key_iv: "aa590f21f11bd75bd0f8d97a74a83082"
server_key_version: "encryptionkey@0"
I20260925 22:49:03.084798 32505 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260925 22:49:03.088011 32520 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:49:03.089603 32505 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 22:49:03.089828 32505 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790376259650642-11032-0/minicluster-data/master-0/data,/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790376259650642-11032-0/minicluster-data/master-0/wal
uuid: "51b835858413449380be32af2f930a6b"
format_stamp: "Formatted at 2026-09-25 22:49:03 on dist-test-slave-5vdr"
server_key: "689d7e0ea64315aad666571777886099"
server_key_iv: "aa590f21f11bd75bd0f8d97a74a83082"
server_key_version: "encryptionkey@0"
I20260925 22:49:03.089998 32505 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790376259650642-11032-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790376259650642-11032-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790376259650642-11032-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:49:03.095212 32505 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:49:03.098699 32505 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:49:03.098948 32505 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:49:03.106992 32505 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40251
I20260925 22:49:03.106998 32572 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40251 every 8 connection(s)
I20260925 22:49:03.108144 32505 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790376259650642-11032-0/minicluster-data/master-0/data/info.pb
I20260925 22:49:03.111018 11032 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrWXm8X/build/debug/bin/kudu as pid 32505
I20260925 22:49:03.111155 11032 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790376259650642-11032-0/minicluster-data/master-0/wal/instance
I20260925 22:49:03.111482 11032 external_mini_cluster.cc:1428] Setting key 42b754248c693f80fc4c7d3d5da24ab3
I20260925 22:49:03.111660 32573 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:49:03.117955 32576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:03.113057 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.97:58329 (local address 127.0.0.1:40251)
0925 22:49:03.113651 (+ 594us) server_negotiation.cc:206] Beginning negotiation
0925 22:49:03.113668 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0925 22:49:03.113709 (+ 41us) server_negotiation.cc:407] Connection header received
0925 22:49:03.114807 (+ 1098us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:49:03.114833 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:49:03.115173 (+ 340us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:49:03.115595 (+ 422us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:49:03.116450 (+ 855us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:49:03.116457 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:49:03.116478 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 22:49:03.116489 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:49:03.116548 (+ 59us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 22:49:03.116570 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:49:03.116742 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0925 22:49:03.116821 (+ 79us) server_negotiation.cc:299] Negotiation successful
0925 22:49:03.117012 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":420,"thread_start_us":194,"threads_started":1}
I20260925 22:49:03.119539 32573 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 51b835858413449380be32af2f930a6b: Bootstrap starting.
I20260925 22:49:03.121788 11032 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 22:49:03.122443 32573 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 51b835858413449380be32af2f930a6b: Neither blocks nor log segments found. Creating new log.
I20260925 22:49:03.123368 32573 log.cc:824] T 00000000000000000000000000000000 P 51b835858413449380be32af2f930a6b: Log is configured to *not* fsync() on all Append() calls
I20260925 22:49:03.126349 32573 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 51b835858413449380be32af2f930a6b: No bootstrap required, opened a new log
I20260925 22:49:03.128048 32576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:03.125895 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.97:60595 (local address 127.0.0.1:40251)
0925 22:49:03.126083 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0925 22:49:03.126088 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:49:03.126188 (+ 100us) server_negotiation.cc:407] Connection header received
0925 22:49:03.126316 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:49:03.126319 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:49:03.126359 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:49:03.126479 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:49:03.127558 (+ 1079us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:49:03.127562 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:49:03.127565 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 22:49:03.127579 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:49:03.127612 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 22:49:03.127626 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:49:03.127728 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0925 22:49:03.127826 (+ 98us) server_negotiation.cc:299] Negotiation successful
0925 22:49:03.127898 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260925 22:49:03.131019 32573 raft_consensus.cc:348] T 00000000000000000000000000000000 P 51b835858413449380be32af2f930a6b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "51b835858413449380be32af2f930a6b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40251 } }
I20260925 22:49:03.131357 32573 raft_consensus.cc:374] T 00000000000000000000000000000000 P 51b835858413449380be32af2f930a6b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:49:03.131441 32573 raft_consensus.cc:729] T 00000000000000000000000000000000 P 51b835858413449380be32af2f930a6b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51b835858413449380be32af2f930a6b, State: Initialized, Role: FOLLOWER
I20260925 22:49:03.132007 32573 consensus_queue.cc:260] T 00000000000000000000000000000000 P 51b835858413449380be32af2f930a6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "51b835858413449380be32af2f930a6b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40251 } }
I20260925 22:49:03.132192 32573 raft_consensus.cc:388] T 00000000000000000000000000000000 P 51b835858413449380be32af2f930a6b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 22:49:03.132287 32573 raft_consensus.cc:482] T 00000000000000000000000000000000 P 51b835858413449380be32af2f930a6b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 22:49:03.132432 32573 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 51b835858413449380be32af2f930a6b [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:49:03.133526 32573 raft_consensus.cc:504] T 00000000000000000000000000000000 P 51b835858413449380be32af2f930a6b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "51b835858413449380be32af2f930a6b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40251 } }
I20260925 22:49:03.133961 32573 leader_election.cc:302] T 00000000000000000000000000000000 P 51b835858413449380be32af2f930a6b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 51b835858413449380be32af2f930a6b; no voters:
I20260925 22:49:03.134375 32573 leader_election.cc:288] T 00000000000000000000000000000000 P 51b835858413449380be32af2f930a6b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 22:49:03.134503 32583 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 51b835858413449380be32af2f930a6b [term 1 FOLLOWER]: Leader election won for term 1
I20260925 22:49:03.134826 32583 raft_consensus.cc:686] T 00000000000000000000000000000000 P 51b835858413449380be32af2f930a6b [term 1 LEADER]: Becoming Leader. State: Replica: 51b835858413449380be32af2f930a6b, State: Running, Role: LEADER
I20260925 22:49:03.135254 32583 consensus_queue.cc:237] T 00000000000000000000000000000000 P 51b835858413449380be32af2f930a6b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "51b835858413449380be32af2f930a6b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40251 } }
I20260925 22:49:03.135957 32573 sys_catalog.cc:564] T 00000000000000000000000000000000 P 51b835858413449380be32af2f930a6b [sys.catalog]: configured and running, proceeding with master startup.
I20260925 22:49:03.137619 32585 sys_catalog.cc:455] T 00000000000000000000000000000000 P 51b835858413449380be32af2f930a6b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "51b835858413449380be32af2f930a6b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "51b835858413449380be32af2f930a6b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40251 } } }
I20260925 22:49:03.137629 32584 sys_catalog.cc:455] T 00000000000000000000000000000000 P 51b835858413449380be32af2f930a6b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 51b835858413449380be32af2f930a6b. Latest consensus state: current_term: 1 leader_uuid: "51b835858413449380be32af2f930a6b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "51b835858413449380be32af2f930a6b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40251 } } }
I20260925 22:49:03.137867 32584 sys_catalog.cc:458] T 00000000000000000000000000000000 P 51b835858413449380be32af2f930a6b [sys.catalog]: This master's current role is: LEADER
I20260925 22:49:03.138322 32585 sys_catalog.cc:458] T 00000000000000000000000000000000 P 51b835858413449380be32af2f930a6b [sys.catalog]: This master's current role is: LEADER
W20260925 22:49:03.141301 32597 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 51b835858413449380be32af2f930a6b: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 22:49:03.141464 32597 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 22:49:03.141564 32598 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 22:49:03.142321 32598 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 22:49:03.149948 32598 catalog_manager.cc:1347] Generated new cluster ID: 52db3175304043ddbce7f071b0d99762
I20260925 22:49:03.150044 32598 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 22:49:03.177835 32598 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 22:49:03.178220 32598 catalog_manager.cc:1504] Loading token signing keys...
I20260925 22:49:03.186220 32598 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 51b835858413449380be32af2f930a6b: Generated new TSK 0
I20260925 22:49:03.187213 32598 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 22:49:03.191059 11032 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrWXm8X/build/debug/bin/kudu with pid 32505
2026-09-25T22:49:03Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1449 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790376259650642-11032-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 22:49:03 dist-test-slave-5vdr krb5kdc[32604](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 22:49:03 dist-test-slave-5vdr krb5kdc[32604](info): set up 2 sockets
Sep 25 22:49:03 dist-test-slave-5vdr krb5kdc[32604](info): commencing operation
krb5kdc: starting...
W20260925 22:49:04.244289 11032 mini_kdc.cc:121] Time spent starting KDC: real 1.032s user 0.004s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790376259650642-11032-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790376259650642-11032-0/krb5kdc/test-user.keytab.
Sep 25 22:49:04 dist-test-slave-5vdr krb5kdc[32604](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376544, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T22:49:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T22:49:04Z Disabled control of system clock
WARNING: no policy specified for kudu/127.10.198.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.198.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.198.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790376259650642-11032-0/krb5kdc/kudu_127.10.198.62.keytab.
Entry for principal kudu/127.10.198.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790376259650642-11032-0/krb5kdc/kudu_127.10.198.62.keytab.
I20260925 22:49:04.375144 11032 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrWXm8X/build/debug/bin/kudu
/tmp/dist-test-taskrWXm8X/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790376259650642-11032-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790376259650642-11032-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790376259650642-11032-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790376259650642-11032-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.198.62:39651
--webserver_interface=127.10.198.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790376259650642-11032-0/krb5kdc/kudu_127.10.198.62.keytab
--principal=kudu/127.10.198.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.198.20:39543
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.198.62:39651
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790376259650642-11032-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790376259650642-11032-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790376259650642-11032-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:49:04.495641 32617 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:49:04.496134 32617 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:49:04.496232 32617 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:49:04.500658 32617 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 22:49:04.500790 32617 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:49:04.500828 32617 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:49:04.500870 32617 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 22:49:04.500912 32617 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 22:49:04.507516 32617 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.198.20:39543
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790376259650642-11032-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790376259650642-11032-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.198.62:39651
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790376259650642-11032-0/krb5kdc/kudu_127.10.198.62.keytab
--principal=kudu/127.10.198.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.198.62:39651
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790376259650642-11032-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.198.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32617
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790376259650642-11032-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:09:16 UTC on ccycloud.quasar-afmhyw.root.comops.site
build id 4673379
I20260925 22:49:04.508829 32617 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:49:04.509994 32617 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:49:04.519934 32625 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:49:04.520004 32623 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:49:04.520073 32622 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:49:04.520340 32617 server_base.cc:1034] running on GCE node
I20260925 22:49:04.521086 32617 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:49:04.522480 32617 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:49:04.523716 32617 hybrid_clock.cc:648] HybridClock initialized: now 1790376544523688 us; error 56 us; skew 500 ppm
Sep 25 22:49:04 dist-test-slave-5vdr krb5kdc[32604](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376544, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.198.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:49:04.527068 32617 init.cc:377] Logged in from keytab as kudu/127.10.198.62@KRBTEST.COM (short username kudu)
I20260925 22:49:04.528285 32617 webserver.cc:460] Webserver started at http://127.10.198.62:42515/ using document root <none> and password file <none>
I20260925 22:49:04.528951 32617 fs_manager.cc:359] Metadata directory not provided
I20260925 22:49:04.529062 32617 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:49:04.529335 32617 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:49:04.532466 32617 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790376259650642-11032-0/minicluster-data/master-0/data/instance:
uuid: "f255d346ee134b13844709e9b859fbfa"
format_stamp: "Formatted at 2026-09-25 22:49:04 on dist-test-slave-5vdr"
server_key: "3fa7e5c8fb194d864f5533b4a3c45273"
server_key_iv: "8c8699ee06282239dfdf99e00145f7b8"
server_key_version: "encryptionkey@0"
I20260925 22:49:04.533749 32617 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790376259650642-11032-0/minicluster-data/master-0/wal/instance:
uuid: "f255d346ee134b13844709e9b859fbfa"
format_stamp: "Formatted at 2026-09-25 22:49:04 on dist-test-slave-5vdr"
server_key: "3fa7e5c8fb194d864f5533b4a3c45273"
server_key_iv: "8c8699ee06282239dfdf99e00145f7b8"
server_key_version: "encryptionkey@0"
I20260925 22:49:04.538522 32617 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260925 22:49:04.541821 32632 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:49:04.543674 32617 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260925 22:49:04.543977 32617 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790376259650642-11032-0/minicluster-data/master-0/data,/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790376259650642-11032-0/minicluster-data/master-0/wal
uuid: "f255d346ee134b13844709e9b859fbfa"
format_stamp: "Formatted at 2026-09-25 22:49:04 on dist-test-slave-5vdr"
server_key: "3fa7e5c8fb194d864f5533b4a3c45273"
server_key_iv: "8c8699ee06282239dfdf99e00145f7b8"
server_key_version: "encryptionkey@0"
I20260925 22:49:04.544163 32617 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790376259650642-11032-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790376259650642-11032-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790376259650642-11032-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:49:04.569188 32617 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:49:04.572573 32617 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:49:04.572834 32617 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:49:04.580456 32617 rpc_server.cc:307] RPC server started. Bound to: 127.10.198.62:39651
I20260925 22:49:04.580483 32684 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.198.62:39651 every 8 connection(s)
I20260925 22:49:04.581571 32617 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790376259650642-11032-0/minicluster-data/master-0/data/info.pb
I20260925 22:49:04.583236 11032 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrWXm8X/build/debug/bin/kudu as pid 32617
I20260925 22:49:04.583389 11032 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790376259650642-11032-0/minicluster-data/master-0/wal/instance
I20260925 22:49:04.583727 11032 external_mini_cluster.cc:1428] Setting key 158dcfe2d13367ac657f199e89ee7859
I20260925 22:49:04.585160 32685 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:49:04.591485 32685 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f255d346ee134b13844709e9b859fbfa: Bootstrap starting.
Sep 25 22:49:04 dist-test-slave-5vdr krb5kdc[32604](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376544, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.198.62@KRBTEST.COM
I20260925 22:49:04.594605 32685 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f255d346ee134b13844709e9b859fbfa: Neither blocks nor log segments found. Creating new log.
I20260925 22:49:04.595793 32685 log.cc:824] T 00000000000000000000000000000000 P f255d346ee134b13844709e9b859fbfa: Log is configured to *not* fsync() on all Append() calls
I20260925 22:49:04.599149 32685 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f255d346ee134b13844709e9b859fbfa: No bootstrap required, opened a new log
I20260925 22:49:04.600643 32688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:04.585338 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52829 (local address 127.10.198.62:39651)
0925 22:49:04.585859 (+ 521us) server_negotiation.cc:206] Beginning negotiation
0925 22:49:04.585875 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0925 22:49:04.585936 (+ 61us) server_negotiation.cc:407] Connection header received
0925 22:49:04.586636 (+ 700us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:49:04.586655 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:49:04.587044 (+ 389us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:49:04.587444 (+ 400us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:49:04.588427 (+ 983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:49:04.589483 (+ 1056us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:49:04.590473 (+ 990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:49:04.591007 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:49:04.591205 (+ 198us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:49:04.594297 (+ 3092us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:49:04.594305 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:49:04.594325 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:49:04.594367 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:49:04.596698 (+ 2331us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:49:04.597660 (+ 962us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:49:04.597667 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:49:04.597675 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:49:04.597836 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:49:04.598232 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:49:04.598236 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:49:04.598237 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:49:04.598795 (+ 558us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:49:04.599030 (+ 235us) server_negotiation.cc:1035] Waiting for connection context
0925 22:49:04.599346 (+ 316us) server_negotiation.cc:299] Negotiation successful
0925 22:49:04.599609 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":364,"thread_start_us":168,"threads_started":1}
I20260925 22:49:04.603820 32685 raft_consensus.cc:348] T 00000000000000000000000000000000 P f255d346ee134b13844709e9b859fbfa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f255d346ee134b13844709e9b859fbfa" member_type: VOTER last_known_addr { host: "127.10.198.62" port: 39651 } }
I20260925 22:49:04.604221 32685 raft_consensus.cc:374] T 00000000000000000000000000000000 P f255d346ee134b13844709e9b859fbfa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:49:04.604274 32685 raft_consensus.cc:729] T 00000000000000000000000000000000 P f255d346ee134b13844709e9b859fbfa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f255d346ee134b13844709e9b859fbfa, State: Initialized, Role: FOLLOWER
I20260925 22:49:04.604660 11032 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 22:49:04.604799 32685 consensus_queue.cc:260] T 00000000000000000000000000000000 P f255d346ee134b13844709e9b859fbfa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f255d346ee134b13844709e9b859fbfa" member_type: VOTER last_known_addr { host: "127.10.198.62" port: 39651 } }
I20260925 22:49:04.604978 32685 raft_consensus.cc:388] T 00000000000000000000000000000000 P f255d346ee134b13844709e9b859fbfa [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 22:49:04.605048 32685 raft_consensus.cc:482] T 00000000000000000000000000000000 P f255d346ee134b13844709e9b859fbfa [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 22:49:04.605155 32685 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f255d346ee134b13844709e9b859fbfa [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:49:04.606328 32685 raft_consensus.cc:504] T 00000000000000000000000000000000 P f255d346ee134b13844709e9b859fbfa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f255d346ee134b13844709e9b859fbfa" member_type: VOTER last_known_addr { host: "127.10.198.62" port: 39651 } }
I20260925 22:49:04.606748 32685 leader_election.cc:302] T 00000000000000000000000000000000 P f255d346ee134b13844709e9b859fbfa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f255d346ee134b13844709e9b859fbfa; no voters:
I20260925 22:49:04.607158 32685 leader_election.cc:288] T 00000000000000000000000000000000 P f255d346ee134b13844709e9b859fbfa [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 22:49:04.607223 32695 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f255d346ee134b13844709e9b859fbfa [term 1 FOLLOWER]: Leader election won for term 1
I20260925 22:49:04.607429 32695 raft_consensus.cc:686] T 00000000000000000000000000000000 P f255d346ee134b13844709e9b859fbfa [term 1 LEADER]: Becoming Leader. State: Replica: f255d346ee134b13844709e9b859fbfa, State: Running, Role: LEADER
I20260925 22:49:04.607823 32695 consensus_queue.cc:237] T 00000000000000000000000000000000 P f255d346ee134b13844709e9b859fbfa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f255d346ee134b13844709e9b859fbfa" member_type: VOTER last_known_addr { host: "127.10.198.62" port: 39651 } }
I20260925 22:49:04.608664 32685 sys_catalog.cc:564] T 00000000000000000000000000000000 P f255d346ee134b13844709e9b859fbfa [sys.catalog]: configured and running, proceeding with master startup.
I20260925 22:49:04.610082 32696 sys_catalog.cc:455] T 00000000000000000000000000000000 P f255d346ee134b13844709e9b859fbfa [sys.catalog]: SysCatalogTable state changed. Reason: New leader f255d346ee134b13844709e9b859fbfa. Latest consensus state: current_term: 1 leader_uuid: "f255d346ee134b13844709e9b859fbfa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f255d346ee134b13844709e9b859fbfa" member_type: VOTER last_known_addr { host: "127.10.198.62" port: 39651 } } }
I20260925 22:49:04.610236 32696 sys_catalog.cc:458] T 00000000000000000000000000000000 P f255d346ee134b13844709e9b859fbfa [sys.catalog]: This master's current role is: LEADER
I20260925 22:49:04.610574 32697 sys_catalog.cc:455] T 00000000000000000000000000000000 P f255d346ee134b13844709e9b859fbfa [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f255d346ee134b13844709e9b859fbfa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f255d346ee134b13844709e9b859fbfa" member_type: VOTER last_known_addr { host: "127.10.198.62" port: 39651 } } }
I20260925 22:49:04.610703 32706 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 22:49:04.610778 32697 sys_catalog.cc:458] T 00000000000000000000000000000000 P f255d346ee134b13844709e9b859fbfa [sys.catalog]: This master's current role is: LEADER
I20260925 22:49:04.614594 32706 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 22:49:04.617024 32688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:04.607019 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34969 (local address 127.10.198.62:39651)
0925 22:49:04.607169 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0925 22:49:04.607175 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 22:49:04.607462 (+ 287us) server_negotiation.cc:407] Connection header received
0925 22:49:04.607678 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:49:04.607683 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:49:04.607740 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:49:04.607796 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:49:04.608705 (+ 909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:49:04.609466 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:49:04.610709 (+ 1243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:49:04.611092 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:49:04.611202 (+ 110us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:49:04.613040 (+ 1838us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:49:04.613045 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:49:04.613047 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:49:04.613082 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:49:04.614896 (+ 1814us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:49:04.615543 (+ 647us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:49:04.615548 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:49:04.615550 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:49:04.615643 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:49:04.616067 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:49:04.616070 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:49:04.616072 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:49:04.616272 (+ 200us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:49:04.616440 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0925 22:49:04.616746 (+ 306us) server_negotiation.cc:299] Negotiation successful
0925 22:49:04.616849 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260925 22:49:04.622167 32706 catalog_manager.cc:1347] Generated new cluster ID: 692280848525438fb4aee22ddbbc0fc2
I20260925 22:49:04.622267 32706 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 22:49:04.641804 32706 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 22:49:04.642879 32706 catalog_manager.cc:1504] Loading token signing keys...
I20260925 22:49:04.648922 32706 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f255d346ee134b13844709e9b859fbfa: Generated new TSK 0
I20260925 22:49:04.649925 32706 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 22:49:04.657651 11032 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrWXm8X/build/debug/bin/kudu with pid 32617
W20260925 22:49:04.663223 32602 connection.cc:535] client connection to 127.10.198.62:39651 recv error: Network error: failed to read from TLS socket (remote: 127.10.198.62:39651): Connection reset by peer (error 104)
2026-09-25T22:49:04Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (1465 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790376259650642-11032-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 22:49:04 dist-test-slave-5vdr krb5kdc[32716](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 22:49:04 dist-test-slave-5vdr krb5kdc[32716](info): set up 2 sockets
Sep 25 22:49:04 dist-test-slave-5vdr krb5kdc[32716](info): commencing operation
krb5kdc: starting...
W20260925 22:49:05.784641 11032 mini_kdc.cc:121] Time spent starting KDC: real 1.107s user 0.001s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790376259650642-11032-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790376259650642-11032-0/krb5kdc/test-user.keytab.
Sep 25 22:49:05 dist-test-slave-5vdr krb5kdc[32716](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376545, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T22:49:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T22:49:05Z Disabled control of system clock
WARNING: no policy specified for kudu/127.10.198.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.198.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.198.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790376259650642-11032-0/krb5kdc/kudu_127.10.198.62.keytab.
Entry for principal kudu/127.10.198.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790376259650642-11032-0/krb5kdc/kudu_127.10.198.62.keytab.
I20260925 22:49:05.919390 11032 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrWXm8X/build/debug/bin/kudu
/tmp/dist-test-taskrWXm8X/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790376259650642-11032-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790376259650642-11032-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790376259650642-11032-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790376259650642-11032-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.198.62:40411
--webserver_interface=127.10.198.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790376259650642-11032-0/krb5kdc/kudu_127.10.198.62.keytab
--principal=kudu/127.10.198.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.198.20:45141
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.198.62:40411
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790376259650642-11032-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790376259650642-11032-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790376259650642-11032-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:49:06.039549 32729 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:49:06.039992 32729 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:49:06.040084 32729 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:49:06.044457 32729 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 22:49:06.044601 32729 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:49:06.044638 32729 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:49:06.044683 32729 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 22:49:06.044744 32729 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 22:49:06.050689 32729 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.198.20:45141
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790376259650642-11032-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790376259650642-11032-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.198.62:40411
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790376259650642-11032-0/krb5kdc/kudu_127.10.198.62.keytab
--principal=kudu/127.10.198.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.198.62:40411
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790376259650642-11032-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.198.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32729
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790376259650642-11032-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:09:16 UTC on ccycloud.quasar-afmhyw.root.comops.site
build id 4673379
I20260925 22:49:06.052047 32729 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:49:06.053229 32729 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:49:06.062794 32737 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:49:06.062799 32735 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:49:06.062794 32734 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:49:06.063181 32729 server_base.cc:1034] running on GCE node
I20260925 22:49:06.064045 32729 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:49:06.065400 32729 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:49:06.066650 32729 hybrid_clock.cc:648] HybridClock initialized: now 1790376546066630 us; error 42 us; skew 500 ppm
Sep 25 22:49:06 dist-test-slave-5vdr krb5kdc[32716](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376546, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.198.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:49:06.070043 32729 init.cc:377] Logged in from keytab as kudu/127.10.198.62@KRBTEST.COM (short username kudu)
I20260925 22:49:06.071233 32729 webserver.cc:460] Webserver started at http://127.10.198.62:38353/ using document root <none> and password file <none>
I20260925 22:49:06.071975 32729 fs_manager.cc:359] Metadata directory not provided
I20260925 22:49:06.072086 32729 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:49:06.072348 32729 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:49:06.074743 32729 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790376259650642-11032-0/minicluster-data/master-0/data/instance:
uuid: "aa2b87daa49d436a8700b3959411fc69"
format_stamp: "Formatted at 2026-09-25 22:49:06 on dist-test-slave-5vdr"
server_key: "455be9a187627fbea49fd4fdba836bf4"
server_key_iv: "064f0fa968cb3859a95ae68648e5f67e"
server_key_version: "encryptionkey@0"
I20260925 22:49:06.075460 32729 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790376259650642-11032-0/minicluster-data/master-0/wal/instance:
uuid: "aa2b87daa49d436a8700b3959411fc69"
format_stamp: "Formatted at 2026-09-25 22:49:06 on dist-test-slave-5vdr"
server_key: "455be9a187627fbea49fd4fdba836bf4"
server_key_iv: "064f0fa968cb3859a95ae68648e5f67e"
server_key_version: "encryptionkey@0"
I20260925 22:49:06.080238 32729 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260925 22:49:06.083519 32744 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:49:06.085377 32729 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 22:49:06.085597 32729 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790376259650642-11032-0/minicluster-data/master-0/data,/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790376259650642-11032-0/minicluster-data/master-0/wal
uuid: "aa2b87daa49d436a8700b3959411fc69"
format_stamp: "Formatted at 2026-09-25 22:49:06 on dist-test-slave-5vdr"
server_key: "455be9a187627fbea49fd4fdba836bf4"
server_key_iv: "064f0fa968cb3859a95ae68648e5f67e"
server_key_version: "encryptionkey@0"
I20260925 22:49:06.085767 32729 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790376259650642-11032-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790376259650642-11032-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790376259650642-11032-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:49:06.108255 32729 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:49:06.111897 32729 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:49:06.112164 32729 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:49:06.120986 32729 rpc_server.cc:307] RPC server started. Bound to: 127.10.198.62:40411
I20260925 22:49:06.120998 328 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.198.62:40411 every 8 connection(s)
I20260925 22:49:06.122150 32729 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790376259650642-11032-0/minicluster-data/master-0/data/info.pb
I20260925 22:49:06.125432 329 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:49:06.127317 11032 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrWXm8X/build/debug/bin/kudu as pid 32729
I20260925 22:49:06.127503 11032 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790376259650642-11032-0/minicluster-data/master-0/wal/instance
I20260925 22:49:06.127815 11032 external_mini_cluster.cc:1428] Setting key 6f71c38bad4855948eb5fed790a941de
I20260925 22:49:06.133425 329 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aa2b87daa49d436a8700b3959411fc69: Bootstrap starting.
I20260925 22:49:06.136328 329 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P aa2b87daa49d436a8700b3959411fc69: Neither blocks nor log segments found. Creating new log.
I20260925 22:49:06.137217 329 log.cc:824] T 00000000000000000000000000000000 P aa2b87daa49d436a8700b3959411fc69: Log is configured to *not* fsync() on all Append() calls
I20260925 22:49:06.138656 332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:06.129706 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58477 (local address 127.10.198.62:40411)
0925 22:49:06.130274 (+ 568us) server_negotiation.cc:206] Beginning negotiation
0925 22:49:06.130291 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0925 22:49:06.130346 (+ 55us) server_negotiation.cc:407] Connection header received
0925 22:49:06.131058 (+ 712us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:49:06.131077 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:49:06.131477 (+ 400us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:49:06.131966 (+ 489us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:49:06.132435 (+ 469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:49:06.133510 (+ 1075us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:49:06.134567 (+ 1057us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:49:06.135127 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:49:06.135332 (+ 205us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:49:06.136430 (+ 1098us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:49:06.136436 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:49:06.136453 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 22:49:06.136973 (+ 520us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:49:06.137045 (+ 72us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 22:49:06.137080 (+ 35us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:49:06.137246 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0925 22:49:06.137468 (+ 222us) server_negotiation.cc:299] Negotiation successful
0925 22:49:06.137631 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":396,"thread_start_us":179,"threads_started":1}
I20260925 22:49:06.139935 329 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aa2b87daa49d436a8700b3959411fc69: No bootstrap required, opened a new log
I20260925 22:49:06.142319 11032 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 22:49:06.144433 329 raft_consensus.cc:348] T 00000000000000000000000000000000 P aa2b87daa49d436a8700b3959411fc69 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa2b87daa49d436a8700b3959411fc69" member_type: VOTER last_known_addr { host: "127.10.198.62" port: 40411 } }
I20260925 22:49:06.144876 329 raft_consensus.cc:374] T 00000000000000000000000000000000 P aa2b87daa49d436a8700b3959411fc69 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:49:06.145012 329 raft_consensus.cc:729] T 00000000000000000000000000000000 P aa2b87daa49d436a8700b3959411fc69 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aa2b87daa49d436a8700b3959411fc69, State: Initialized, Role: FOLLOWER
I20260925 22:49:06.145632 329 consensus_queue.cc:260] T 00000000000000000000000000000000 P aa2b87daa49d436a8700b3959411fc69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa2b87daa49d436a8700b3959411fc69" member_type: VOTER last_known_addr { host: "127.10.198.62" port: 40411 } }
I20260925 22:49:06.145854 329 raft_consensus.cc:388] T 00000000000000000000000000000000 P aa2b87daa49d436a8700b3959411fc69 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 22:49:06.145944 329 raft_consensus.cc:482] T 00000000000000000000000000000000 P aa2b87daa49d436a8700b3959411fc69 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 22:49:06.146091 329 raft_consensus.cc:3037] T 00000000000000000000000000000000 P aa2b87daa49d436a8700b3959411fc69 [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:49:06.147449 329 raft_consensus.cc:504] T 00000000000000000000000000000000 P aa2b87daa49d436a8700b3959411fc69 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa2b87daa49d436a8700b3959411fc69" member_type: VOTER last_known_addr { host: "127.10.198.62" port: 40411 } }
I20260925 22:49:06.148005 329 leader_election.cc:302] T 00000000000000000000000000000000 P aa2b87daa49d436a8700b3959411fc69 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: aa2b87daa49d436a8700b3959411fc69; no voters:
I20260925 22:49:06.148468 329 leader_election.cc:288] T 00000000000000000000000000000000 P aa2b87daa49d436a8700b3959411fc69 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 22:49:06.148617 339 raft_consensus.cc:2781] T 00000000000000000000000000000000 P aa2b87daa49d436a8700b3959411fc69 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 22:49:06.148941 339 raft_consensus.cc:686] T 00000000000000000000000000000000 P aa2b87daa49d436a8700b3959411fc69 [term 1 LEADER]: Becoming Leader. State: Replica: aa2b87daa49d436a8700b3959411fc69, State: Running, Role: LEADER
I20260925 22:49:06.149435 339 consensus_queue.cc:237] T 00000000000000000000000000000000 P aa2b87daa49d436a8700b3959411fc69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa2b87daa49d436a8700b3959411fc69" member_type: VOTER last_known_addr { host: "127.10.198.62" port: 40411 } }
I20260925 22:49:06.149999 332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:06.144852 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42195 (local address 127.10.198.62:40411)
0925 22:49:06.145013 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0925 22:49:06.145020 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 22:49:06.145205 (+ 185us) server_negotiation.cc:407] Connection header received
0925 22:49:06.145382 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:49:06.145385 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:49:06.145446 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:49:06.145521 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:49:06.145889 (+ 368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:49:06.146568 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:49:06.147766 (+ 1198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:49:06.148202 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:49:06.148339 (+ 137us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:49:06.149393 (+ 1054us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:49:06.149397 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:49:06.149400 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 22:49:06.149419 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:49:06.149460 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 22:49:06.149479 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:49:06.149589 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0925 22:49:06.149817 (+ 228us) server_negotiation.cc:299] Negotiation successful
0925 22:49:06.149861 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260925 22:49:06.150099 329 sys_catalog.cc:564] T 00000000000000000000000000000000 P aa2b87daa49d436a8700b3959411fc69 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 22:49:06.151543 341 sys_catalog.cc:455] T 00000000000000000000000000000000 P aa2b87daa49d436a8700b3959411fc69 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "aa2b87daa49d436a8700b3959411fc69" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa2b87daa49d436a8700b3959411fc69" member_type: VOTER last_known_addr { host: "127.10.198.62" port: 40411 } } }
I20260925 22:49:06.151722 341 sys_catalog.cc:458] T 00000000000000000000000000000000 P aa2b87daa49d436a8700b3959411fc69 [sys.catalog]: This master's current role is: LEADER
I20260925 22:49:06.152084 340 sys_catalog.cc:455] T 00000000000000000000000000000000 P aa2b87daa49d436a8700b3959411fc69 [sys.catalog]: SysCatalogTable state changed. Reason: New leader aa2b87daa49d436a8700b3959411fc69. Latest consensus state: current_term: 1 leader_uuid: "aa2b87daa49d436a8700b3959411fc69" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa2b87daa49d436a8700b3959411fc69" member_type: VOTER last_known_addr { host: "127.10.198.62" port: 40411 } } }
I20260925 22:49:06.152204 340 sys_catalog.cc:458] T 00000000000000000000000000000000 P aa2b87daa49d436a8700b3959411fc69 [sys.catalog]: This master's current role is: LEADER
I20260925 22:49:06.152593 350 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 22:49:06.156342 350 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 22:49:06.162920 350 catalog_manager.cc:1347] Generated new cluster ID: 2c344078f1eb48248e4972c8f8913920
I20260925 22:49:06.163125 350 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 22:49:06.190269 350 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 22:49:06.191409 350 catalog_manager.cc:1504] Loading token signing keys...
I20260925 22:49:06.197855 350 catalog_manager.cc:5868] T 00000000000000000000000000000000 P aa2b87daa49d436a8700b3959411fc69: Generated new TSK 0
I20260925 22:49:06.198671 350 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 22:49:06.209933 11032 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrWXm8X/build/debug/bin/kudu with pid 32729
W20260925 22:49:06.215514 32714 connection.cc:535] client connection to 127.10.198.62:40411 recv error: Network error: failed to read from TLS socket (remote: 127.10.198.62:40411): Connection reset by peer (error 104)
2026-09-25T22:49:06Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1552 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790376259650642-11032-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 22:49:06 dist-test-slave-5vdr krb5kdc[360](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 22:49:06 dist-test-slave-5vdr krb5kdc[360](info): set up 2 sockets
Sep 25 22:49:06 dist-test-slave-5vdr krb5kdc[360](info): commencing operation
krb5kdc: starting...
W20260925 22:49:07.264760 11032 mini_kdc.cc:121] Time spent starting KDC: real 1.034s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790376259650642-11032-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790376259650642-11032-0/krb5kdc/test-user.keytab.
Sep 25 22:49:07 dist-test-slave-5vdr krb5kdc[360](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376547, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T22:49:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T22:49:07Z Disabled control of system clock
WARNING: no policy specified for kudu/127.10.198.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.198.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.198.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790376259650642-11032-0/krb5kdc/kudu_127.10.198.62.keytab.
Entry for principal kudu/127.10.198.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790376259650642-11032-0/krb5kdc/kudu_127.10.198.62.keytab.
I20260925 22:49:07.395634 11032 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrWXm8X/build/debug/bin/kudu
/tmp/dist-test-taskrWXm8X/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790376259650642-11032-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790376259650642-11032-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790376259650642-11032-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790376259650642-11032-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.198.62:41621
--webserver_interface=127.10.198.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790376259650642-11032-0/krb5kdc/kudu_127.10.198.62.keytab
--principal=kudu/127.10.198.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.198.20:37987
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.198.62:41621
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790376259650642-11032-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790376259650642-11032-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790376259650642-11032-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:49:07.520628 373 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:49:07.520956 373 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:49:07.521003 373 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:49:07.525506 373 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 22:49:07.525601 373 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:49:07.525622 373 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:49:07.525640 373 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 22:49:07.525657 373 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 22:49:07.532059 373 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.198.20:37987
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790376259650642-11032-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790376259650642-11032-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.198.62:41621
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790376259650642-11032-0/krb5kdc/kudu_127.10.198.62.keytab
--principal=kudu/127.10.198.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.198.62:41621
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790376259650642-11032-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.198.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.373
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790376259650642-11032-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:09:16 UTC on ccycloud.quasar-afmhyw.root.comops.site
build id 4673379
I20260925 22:49:07.533360 373 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:49:07.534688 373 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:49:07.545331 381 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:49:07.545333 378 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:49:07.545398 373 server_base.cc:1034] running on GCE node
W20260925 22:49:07.545331 379 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:49:07.546221 373 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:49:07.547693 373 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:49:07.548923 373 hybrid_clock.cc:648] HybridClock initialized: now 1790376547548884 us; error 62 us; skew 500 ppm
Sep 25 22:49:07 dist-test-slave-5vdr krb5kdc[360](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376547, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.198.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:49:07.552414 373 init.cc:377] Logged in from keytab as kudu/127.10.198.62@KRBTEST.COM (short username kudu)
I20260925 22:49:07.553592 373 webserver.cc:460] Webserver started at http://127.10.198.62:38107/ using document root <none> and password file <none>
I20260925 22:49:07.554230 373 fs_manager.cc:359] Metadata directory not provided
I20260925 22:49:07.554334 373 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:49:07.554600 373 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:49:07.557009 373 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790376259650642-11032-0/minicluster-data/master-0/data/instance:
uuid: "cf3cf75c14b840f88a2eb7a07a26c579"
format_stamp: "Formatted at 2026-09-25 22:49:07 on dist-test-slave-5vdr"
server_key: "121ca50e64a6933ff9f932347a75a8b6"
server_key_iv: "df15e41a87718eced277f6c69045ba12"
server_key_version: "encryptionkey@0"
I20260925 22:49:07.557664 373 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790376259650642-11032-0/minicluster-data/master-0/wal/instance:
uuid: "cf3cf75c14b840f88a2eb7a07a26c579"
format_stamp: "Formatted at 2026-09-25 22:49:07 on dist-test-slave-5vdr"
server_key: "121ca50e64a6933ff9f932347a75a8b6"
server_key_iv: "df15e41a87718eced277f6c69045ba12"
server_key_version: "encryptionkey@0"
I20260925 22:49:07.562052 373 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.004s
I20260925 22:49:07.565088 388 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:49:07.566514 373 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260925 22:49:07.566709 373 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790376259650642-11032-0/minicluster-data/master-0/data,/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790376259650642-11032-0/minicluster-data/master-0/wal
uuid: "cf3cf75c14b840f88a2eb7a07a26c579"
format_stamp: "Formatted at 2026-09-25 22:49:07 on dist-test-slave-5vdr"
server_key: "121ca50e64a6933ff9f932347a75a8b6"
server_key_iv: "df15e41a87718eced277f6c69045ba12"
server_key_version: "encryptionkey@0"
I20260925 22:49:07.566861 373 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790376259650642-11032-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790376259650642-11032-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790376259650642-11032-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:49:07.572453 373 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:49:07.576737 373 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:49:07.577003 373 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:49:07.585644 373 rpc_server.cc:307] RPC server started. Bound to: 127.10.198.62:41621
I20260925 22:49:07.585667 440 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.198.62:41621 every 8 connection(s)
I20260925 22:49:07.586737 373 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790376259650642-11032-0/minicluster-data/master-0/data/info.pb
I20260925 22:49:07.589895 441 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:49:07.593255 11032 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrWXm8X/build/debug/bin/kudu as pid 373
I20260925 22:49:07.593417 11032 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790376259650642-11032-0/minicluster-data/master-0/wal/instance
I20260925 22:49:07.593747 11032 external_mini_cluster.cc:1428] Setting key 38368f244e8cb915d3d3181e505f829c
I20260925 22:49:07.596912 441 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cf3cf75c14b840f88a2eb7a07a26c579: Bootstrap starting.
I20260925 22:49:07.599987 441 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cf3cf75c14b840f88a2eb7a07a26c579: Neither blocks nor log segments found. Creating new log.
I20260925 22:49:07.600354 444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:07.595470 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38435 (local address 127.10.198.62:41621)
0925 22:49:07.596028 (+ 558us) server_negotiation.cc:206] Beginning negotiation
0925 22:49:07.596048 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0925 22:49:07.596095 (+ 47us) server_negotiation.cc:407] Connection header received
0925 22:49:07.597012 (+ 917us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:49:07.597040 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:49:07.597443 (+ 403us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:49:07.597857 (+ 414us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:49:07.598792 (+ 935us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:49:07.598798 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:49:07.598815 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 22:49:07.598829 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:49:07.598905 (+ 76us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 22:49:07.598932 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:49:07.599080 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0925 22:49:07.599150 (+ 70us) server_negotiation.cc:299] Negotiation successful
0925 22:49:07.599314 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":381,"thread_start_us":152,"threads_started":1}
I20260925 22:49:07.600942 441 log.cc:824] T 00000000000000000000000000000000 P cf3cf75c14b840f88a2eb7a07a26c579: Log is configured to *not* fsync() on all Append() calls
I20260925 22:49:07.603605 441 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cf3cf75c14b840f88a2eb7a07a26c579: No bootstrap required, opened a new log
I20260925 22:49:07.604008 11032 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 22:49:07.608541 444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:07.606513 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58653 (local address 127.10.198.62:41621)
0925 22:49:07.606666 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0925 22:49:07.606672 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 22:49:07.606926 (+ 254us) server_negotiation.cc:407] Connection header received
0925 22:49:07.607101 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:49:07.607105 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:49:07.607151 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:49:07.607242 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:49:07.608079 (+ 837us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:49:07.608082 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:49:07.608085 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 22:49:07.608099 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:49:07.608139 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 22:49:07.608151 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:49:07.608252 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0925 22:49:07.608331 (+ 79us) server_negotiation.cc:299] Negotiation successful
0925 22:49:07.608379 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260925 22:49:07.608500 441 raft_consensus.cc:348] T 00000000000000000000000000000000 P cf3cf75c14b840f88a2eb7a07a26c579 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf3cf75c14b840f88a2eb7a07a26c579" member_type: VOTER last_known_addr { host: "127.10.198.62" port: 41621 } }
I20260925 22:49:07.608877 441 raft_consensus.cc:374] T 00000000000000000000000000000000 P cf3cf75c14b840f88a2eb7a07a26c579 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:49:07.608949 441 raft_consensus.cc:729] T 00000000000000000000000000000000 P cf3cf75c14b840f88a2eb7a07a26c579 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cf3cf75c14b840f88a2eb7a07a26c579, State: Initialized, Role: FOLLOWER
I20260925 22:49:07.609447 441 consensus_queue.cc:260] T 00000000000000000000000000000000 P cf3cf75c14b840f88a2eb7a07a26c579 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf3cf75c14b840f88a2eb7a07a26c579" member_type: VOTER last_known_addr { host: "127.10.198.62" port: 41621 } }
I20260925 22:49:07.609666 441 raft_consensus.cc:388] T 00000000000000000000000000000000 P cf3cf75c14b840f88a2eb7a07a26c579 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 22:49:07.609747 441 raft_consensus.cc:482] T 00000000000000000000000000000000 P cf3cf75c14b840f88a2eb7a07a26c579 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 22:49:07.609859 441 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cf3cf75c14b840f88a2eb7a07a26c579 [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:49:07.610914 441 raft_consensus.cc:504] T 00000000000000000000000000000000 P cf3cf75c14b840f88a2eb7a07a26c579 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf3cf75c14b840f88a2eb7a07a26c579" member_type: VOTER last_known_addr { host: "127.10.198.62" port: 41621 } }
I20260925 22:49:07.611343 441 leader_election.cc:302] T 00000000000000000000000000000000 P cf3cf75c14b840f88a2eb7a07a26c579 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cf3cf75c14b840f88a2eb7a07a26c579; no voters:
I20260925 22:49:07.611711 441 leader_election.cc:288] T 00000000000000000000000000000000 P cf3cf75c14b840f88a2eb7a07a26c579 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 22:49:07.611901 451 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cf3cf75c14b840f88a2eb7a07a26c579 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 22:49:07.612200 451 raft_consensus.cc:686] T 00000000000000000000000000000000 P cf3cf75c14b840f88a2eb7a07a26c579 [term 1 LEADER]: Becoming Leader. State: Replica: cf3cf75c14b840f88a2eb7a07a26c579, State: Running, Role: LEADER
I20260925 22:49:07.612607 451 consensus_queue.cc:237] T 00000000000000000000000000000000 P cf3cf75c14b840f88a2eb7a07a26c579 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf3cf75c14b840f88a2eb7a07a26c579" member_type: VOTER last_known_addr { host: "127.10.198.62" port: 41621 } }
I20260925 22:49:07.613253 441 sys_catalog.cc:564] T 00000000000000000000000000000000 P cf3cf75c14b840f88a2eb7a07a26c579 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 22:49:07.614414 452 sys_catalog.cc:455] T 00000000000000000000000000000000 P cf3cf75c14b840f88a2eb7a07a26c579 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cf3cf75c14b840f88a2eb7a07a26c579" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf3cf75c14b840f88a2eb7a07a26c579" member_type: VOTER last_known_addr { host: "127.10.198.62" port: 41621 } } }
I20260925 22:49:07.614573 452 sys_catalog.cc:458] T 00000000000000000000000000000000 P cf3cf75c14b840f88a2eb7a07a26c579 [sys.catalog]: This master's current role is: LEADER
I20260925 22:49:07.614945 453 sys_catalog.cc:455] T 00000000000000000000000000000000 P cf3cf75c14b840f88a2eb7a07a26c579 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cf3cf75c14b840f88a2eb7a07a26c579. Latest consensus state: current_term: 1 leader_uuid: "cf3cf75c14b840f88a2eb7a07a26c579" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf3cf75c14b840f88a2eb7a07a26c579" member_type: VOTER last_known_addr { host: "127.10.198.62" port: 41621 } } }
I20260925 22:49:07.615129 453 sys_catalog.cc:458] T 00000000000000000000000000000000 P cf3cf75c14b840f88a2eb7a07a26c579 [sys.catalog]: This master's current role is: LEADER
I20260925 22:49:07.616030 460 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 22:49:07.619089 460 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 22:49:07.626124 460 catalog_manager.cc:1347] Generated new cluster ID: 6e4aae21bbe54fa083cdc50e7ab1dd96
I20260925 22:49:07.626258 460 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 22:49:07.637768 460 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 22:49:07.638178 460 catalog_manager.cc:1504] Loading token signing keys...
I20260925 22:49:07.645780 460 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cf3cf75c14b840f88a2eb7a07a26c579: Generated new TSK 0
I20260925 22:49:07.646749 460 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 22:49:07.657018 11032 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrWXm8X/build/debug/bin/kudu with pid 373
2026-09-25T22:49:07Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1446 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790376259650642-11032-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 22:49:07 dist-test-slave-5vdr krb5kdc[472](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 22:49:07 dist-test-slave-5vdr krb5kdc[472](info): set up 2 sockets
Sep 25 22:49:07 dist-test-slave-5vdr krb5kdc[472](info): commencing operation
krb5kdc: starting...
W20260925 22:49:08.744693 11032 mini_kdc.cc:121] Time spent starting KDC: real 1.067s user 0.005s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790376259650642-11032-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790376259650642-11032-0/krb5kdc/test-user.keytab.
Sep 25 22:49:08 dist-test-slave-5vdr krb5kdc[472](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376548, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T22:49:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T22:49:08Z Disabled control of system clock
WARNING: no policy specified for kudu/127.10.198.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.198.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.198.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790376259650642-11032-0/krb5kdc/kudu_127.10.198.62.keytab.
Entry for principal kudu/127.10.198.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790376259650642-11032-0/krb5kdc/kudu_127.10.198.62.keytab.
I20260925 22:49:08.875802 11032 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrWXm8X/build/debug/bin/kudu
/tmp/dist-test-taskrWXm8X/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790376259650642-11032-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790376259650642-11032-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790376259650642-11032-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790376259650642-11032-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.198.62:34939
--webserver_interface=127.10.198.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790376259650642-11032-0/krb5kdc/kudu_127.10.198.62.keytab
--principal=kudu/127.10.198.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.198.20:45045
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.198.62:34939
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790376259650642-11032-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790376259650642-11032-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790376259650642-11032-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:49:08.995152 485 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:49:08.995548 485 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:49:08.995640 485 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:49:09.000098 485 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 22:49:09.000234 485 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:49:09.000269 485 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:49:09.000300 485 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 22:49:09.000334 485 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 22:49:09.006283 485 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.198.20:45045
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790376259650642-11032-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790376259650642-11032-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.198.62:34939
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790376259650642-11032-0/krb5kdc/kudu_127.10.198.62.keytab
--principal=kudu/127.10.198.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.198.62:34939
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790376259650642-11032-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.198.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.485
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790376259650642-11032-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:09:16 UTC on ccycloud.quasar-afmhyw.root.comops.site
build id 4673379
I20260925 22:49:09.007640 485 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:49:09.008780 485 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:49:09.018261 493 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:49:09.018250 491 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:49:09.018250 490 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:49:09.018546 485 server_base.cc:1034] running on GCE node
I20260925 22:49:09.020668 485 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:49:09.022243 485 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:49:09.023645 485 hybrid_clock.cc:648] HybridClock initialized: now 1790376549023551 us; error 133 us; skew 500 ppm
Sep 25 22:49:09 dist-test-slave-5vdr krb5kdc[472](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376549, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.198.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:49:09.026824 485 init.cc:377] Logged in from keytab as kudu/127.10.198.62@KRBTEST.COM (short username kudu)
I20260925 22:49:09.027835 485 webserver.cc:460] Webserver started at http://127.10.198.62:40411/ using document root <none> and password file <none>
I20260925 22:49:09.028512 485 fs_manager.cc:359] Metadata directory not provided
I20260925 22:49:09.028584 485 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:49:09.028764 485 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:49:09.031001 485 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790376259650642-11032-0/minicluster-data/master-0/data/instance:
uuid: "c849c55d71174ff3905506091a4a1dd3"
format_stamp: "Formatted at 2026-09-25 22:49:09 on dist-test-slave-5vdr"
server_key: "df5648ef3da5e5468e4af2e2a43853fc"
server_key_iv: "2275b1a9f4aec86f00ddbc6da61928ae"
server_key_version: "encryptionkey@0"
I20260925 22:49:09.031606 485 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790376259650642-11032-0/minicluster-data/master-0/wal/instance:
uuid: "c849c55d71174ff3905506091a4a1dd3"
format_stamp: "Formatted at 2026-09-25 22:49:09 on dist-test-slave-5vdr"
server_key: "df5648ef3da5e5468e4af2e2a43853fc"
server_key_iv: "2275b1a9f4aec86f00ddbc6da61928ae"
server_key_version: "encryptionkey@0"
I20260925 22:49:09.035938 485 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260925 22:49:09.038781 500 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:49:09.040392 485 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 22:49:09.040535 485 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790376259650642-11032-0/minicluster-data/master-0/data,/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790376259650642-11032-0/minicluster-data/master-0/wal
uuid: "c849c55d71174ff3905506091a4a1dd3"
format_stamp: "Formatted at 2026-09-25 22:49:09 on dist-test-slave-5vdr"
server_key: "df5648ef3da5e5468e4af2e2a43853fc"
server_key_iv: "2275b1a9f4aec86f00ddbc6da61928ae"
server_key_version: "encryptionkey@0"
I20260925 22:49:09.040707 485 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790376259650642-11032-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790376259650642-11032-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790376259650642-11032-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:49:09.054665 485 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:49:09.058458 485 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:49:09.058700 485 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:49:09.066466 485 rpc_server.cc:307] RPC server started. Bound to: 127.10.198.62:34939
I20260925 22:49:09.066550 552 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.198.62:34939 every 8 connection(s)
I20260925 22:49:09.067561 485 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790376259650642-11032-0/minicluster-data/master-0/data/info.pb
I20260925 22:49:09.070689 553 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:49:09.074486 11032 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrWXm8X/build/debug/bin/kudu as pid 485
I20260925 22:49:09.074681 11032 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790376259650642-11032-0/minicluster-data/master-0/wal/instance
I20260925 22:49:09.074957 11032 external_mini_cluster.cc:1428] Setting key f57c62c5178fcf6ca460d8c88e1279d6
I20260925 22:49:09.077644 553 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c849c55d71174ff3905506091a4a1dd3: Bootstrap starting.
I20260925 22:49:09.080678 553 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c849c55d71174ff3905506091a4a1dd3: Neither blocks nor log segments found. Creating new log.
I20260925 22:49:09.081830 553 log.cc:824] T 00000000000000000000000000000000 P c849c55d71174ff3905506091a4a1dd3: Log is configured to *not* fsync() on all Append() calls
I20260925 22:49:09.084574 553 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c849c55d71174ff3905506091a4a1dd3: No bootstrap required, opened a new log
Sep 25 22:49:09 dist-test-slave-5vdr krb5kdc[472](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376548, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.198.62@KRBTEST.COM
I20260925 22:49:09.088552 553 raft_consensus.cc:348] T 00000000000000000000000000000000 P c849c55d71174ff3905506091a4a1dd3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c849c55d71174ff3905506091a4a1dd3" member_type: VOTER last_known_addr { host: "127.10.198.62" port: 34939 } }
I20260925 22:49:09.088856 553 raft_consensus.cc:374] T 00000000000000000000000000000000 P c849c55d71174ff3905506091a4a1dd3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:49:09.088920 553 raft_consensus.cc:729] T 00000000000000000000000000000000 P c849c55d71174ff3905506091a4a1dd3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c849c55d71174ff3905506091a4a1dd3, State: Initialized, Role: FOLLOWER
I20260925 22:49:09.089480 553 consensus_queue.cc:260] T 00000000000000000000000000000000 P c849c55d71174ff3905506091a4a1dd3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c849c55d71174ff3905506091a4a1dd3" member_type: VOTER last_known_addr { host: "127.10.198.62" port: 34939 } }
I20260925 22:49:09.089617 553 raft_consensus.cc:388] T 00000000000000000000000000000000 P c849c55d71174ff3905506091a4a1dd3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 22:49:09.089706 553 raft_consensus.cc:482] T 00000000000000000000000000000000 P c849c55d71174ff3905506091a4a1dd3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 22:49:09.089828 553 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c849c55d71174ff3905506091a4a1dd3 [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:49:09.090953 553 raft_consensus.cc:504] T 00000000000000000000000000000000 P c849c55d71174ff3905506091a4a1dd3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c849c55d71174ff3905506091a4a1dd3" member_type: VOTER last_known_addr { host: "127.10.198.62" port: 34939 } }
I20260925 22:49:09.091341 553 leader_election.cc:302] T 00000000000000000000000000000000 P c849c55d71174ff3905506091a4a1dd3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c849c55d71174ff3905506091a4a1dd3; no voters:
I20260925 22:49:09.091702 553 leader_election.cc:288] T 00000000000000000000000000000000 P c849c55d71174ff3905506091a4a1dd3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 22:49:09.091868 558 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c849c55d71174ff3905506091a4a1dd3 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 22:49:09.092022 556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:09.076504 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52055 (local address 127.10.198.62:34939)
0925 22:49:09.076983 (+ 479us) server_negotiation.cc:206] Beginning negotiation
0925 22:49:09.077003 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0925 22:49:09.077075 (+ 72us) server_negotiation.cc:407] Connection header received
0925 22:49:09.077980 (+ 905us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:49:09.078006 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:49:09.078524 (+ 518us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:49:09.078929 (+ 405us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:49:09.079735 (+ 806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:49:09.081325 (+ 1590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:49:09.082429 (+ 1104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:49:09.083068 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:49:09.083210 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:49:09.086315 (+ 3105us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:49:09.086321 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:49:09.086337 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:49:09.086371 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:49:09.088349 (+ 1978us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:49:09.088881 (+ 532us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:49:09.088895 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:49:09.088901 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:49:09.089100 (+ 199us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:49:09.089419 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:49:09.089424 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:49:09.089426 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:49:09.090051 (+ 625us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:49:09.090293 (+ 242us) server_negotiation.cc:1035] Waiting for connection context
0925 22:49:09.090603 (+ 310us) server_negotiation.cc:299] Negotiation successful
0925 22:49:09.090952 (+ 349us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":318,"thread_start_us":130,"threads_started":1}
I20260925 22:49:09.092133 558 raft_consensus.cc:686] T 00000000000000000000000000000000 P c849c55d71174ff3905506091a4a1dd3 [term 1 LEADER]: Becoming Leader. State: Replica: c849c55d71174ff3905506091a4a1dd3, State: Running, Role: LEADER
I20260925 22:49:09.092687 558 consensus_queue.cc:237] T 00000000000000000000000000000000 P c849c55d71174ff3905506091a4a1dd3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c849c55d71174ff3905506091a4a1dd3" member_type: VOTER last_known_addr { host: "127.10.198.62" port: 34939 } }
I20260925 22:49:09.093250 553 sys_catalog.cc:564] T 00000000000000000000000000000000 P c849c55d71174ff3905506091a4a1dd3 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 22:49:09.094462 560 sys_catalog.cc:455] T 00000000000000000000000000000000 P c849c55d71174ff3905506091a4a1dd3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c849c55d71174ff3905506091a4a1dd3. Latest consensus state: current_term: 1 leader_uuid: "c849c55d71174ff3905506091a4a1dd3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c849c55d71174ff3905506091a4a1dd3" member_type: VOTER last_known_addr { host: "127.10.198.62" port: 34939 } } }
I20260925 22:49:09.094636 560 sys_catalog.cc:458] T 00000000000000000000000000000000 P c849c55d71174ff3905506091a4a1dd3 [sys.catalog]: This master's current role is: LEADER
I20260925 22:49:09.095089 559 sys_catalog.cc:455] T 00000000000000000000000000000000 P c849c55d71174ff3905506091a4a1dd3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c849c55d71174ff3905506091a4a1dd3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c849c55d71174ff3905506091a4a1dd3" member_type: VOTER last_known_addr { host: "127.10.198.62" port: 34939 } } }
I20260925 22:49:09.095285 559 sys_catalog.cc:458] T 00000000000000000000000000000000 P c849c55d71174ff3905506091a4a1dd3 [sys.catalog]: This master's current role is: LEADER
I20260925 22:49:09.096249 11032 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
W20260925 22:49:09.099604 573 catalog_manager.cc:1558] T 00000000000000000000000000000000 P c849c55d71174ff3905506091a4a1dd3: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 22:49:09.099699 573 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 22:49:09.099797 567 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 22:49:09.100948 567 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 22:49:09.108475 567 catalog_manager.cc:1347] Generated new cluster ID: 2a0d48a556164aa1baaf0bf364a4bfbf
I20260925 22:49:09.108682 567 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 22:49:09.109385 556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:09.098799 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53087 (local address 127.10.198.62:34939)
0925 22:49:09.098925 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0925 22:49:09.098929 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:49:09.100306 (+ 1377us) server_negotiation.cc:407] Connection header received
0925 22:49:09.100457 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:49:09.100461 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:49:09.100524 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:49:09.100635 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:49:09.101577 (+ 942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:49:09.102395 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:49:09.103442 (+ 1047us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:49:09.103697 (+ 255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:49:09.103816 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:49:09.105368 (+ 1552us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:49:09.105374 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:49:09.105382 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:49:09.105434 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:49:09.107341 (+ 1907us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:49:09.107985 (+ 644us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:49:09.107988 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:49:09.107990 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:49:09.108079 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:49:09.108412 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:49:09.108415 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:49:09.108416 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:49:09.108665 (+ 249us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:49:09.108817 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0925 22:49:09.109092 (+ 275us) server_negotiation.cc:299] Negotiation successful
0925 22:49:09.109219 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260925 22:49:09.142439 567 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 22:49:09.143544 567 catalog_manager.cc:1504] Loading token signing keys...
I20260925 22:49:09.148485 567 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c849c55d71174ff3905506091a4a1dd3: Generated new TSK 0
I20260925 22:49:09.149381 567 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 22:49:09.158463 11032 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrWXm8X/build/debug/bin/kudu with pid 485
2026-09-25T22:49:09Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (1501 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790376259650642-11032-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 22:49:09 dist-test-slave-5vdr krb5kdc[584](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 22:49:09 dist-test-slave-5vdr krb5kdc[584](info): set up 2 sockets
Sep 25 22:49:09 dist-test-slave-5vdr krb5kdc[584](info): commencing operation
krb5kdc: starting...
W20260925 22:49:10.256305 11032 mini_kdc.cc:121] Time spent starting KDC: real 1.077s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790376259650642-11032-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790376259650642-11032-0/krb5kdc/test-user.keytab.
Sep 25 22:49:10 dist-test-slave-5vdr krb5kdc[584](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376550, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T22:49:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T22:49:10Z Disabled control of system clock
WARNING: no policy specified for kudu/127.10.198.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.198.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.198.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790376259650642-11032-0/krb5kdc/kudu_127.10.198.62.keytab.
Entry for principal kudu/127.10.198.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790376259650642-11032-0/krb5kdc/kudu_127.10.198.62.keytab.
I20260925 22:49:10.391573 11032 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrWXm8X/build/debug/bin/kudu
/tmp/dist-test-taskrWXm8X/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790376259650642-11032-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790376259650642-11032-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790376259650642-11032-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790376259650642-11032-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.198.62:43075
--webserver_interface=127.10.198.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790376259650642-11032-0/krb5kdc/kudu_127.10.198.62.keytab
--principal=kudu/127.10.198.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.198.20:35589
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.198.62:43075
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790376259650642-11032-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790376259650642-11032-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790376259650642-11032-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:49:10.514784 597 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:49:10.515106 597 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:49:10.515151 597 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:49:10.519573 597 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 22:49:10.519665 597 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:49:10.519685 597 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:49:10.519703 597 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 22:49:10.519718 597 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 22:49:10.525810 597 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.198.20:35589
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790376259650642-11032-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790376259650642-11032-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.198.62:43075
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790376259650642-11032-0/krb5kdc/kudu_127.10.198.62.keytab
--principal=kudu/127.10.198.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.198.62:43075
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790376259650642-11032-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.198.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.597
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790376259650642-11032-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:09:16 UTC on ccycloud.quasar-afmhyw.root.comops.site
build id 4673379
I20260925 22:49:10.527068 597 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:49:10.528277 597 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 22:49:10.538245 597 server_base.cc:1034] running on GCE node
W20260925 22:49:10.538309 603 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:49:10.538283 605 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:49:10.538271 602 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:49:10.539220 597 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:49:10.540635 597 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:49:10.541872 597 hybrid_clock.cc:648] HybridClock initialized: now 1790376550541827 us; error 63 us; skew 500 ppm
Sep 25 22:49:10 dist-test-slave-5vdr krb5kdc[584](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376550, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.198.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:49:10.552228 597 init.cc:377] Logged in from keytab as kudu/127.10.198.62@KRBTEST.COM (short username kudu)
I20260925 22:49:10.553380 597 webserver.cc:460] Webserver started at http://127.10.198.62:36421/ using document root <none> and password file <none>
I20260925 22:49:10.554045 597 fs_manager.cc:359] Metadata directory not provided
I20260925 22:49:10.554160 597 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:49:10.554420 597 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:49:10.556840 597 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790376259650642-11032-0/minicluster-data/master-0/data/instance:
uuid: "65fef63633814a5aa8473856092cb9f2"
format_stamp: "Formatted at 2026-09-25 22:49:10 on dist-test-slave-5vdr"
server_key: "9d02f05c776139e8bd778f89cd3d0150"
server_key_iv: "17406a82f99661a14fa589597d75028c"
server_key_version: "encryptionkey@0"
I20260925 22:49:10.557485 597 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790376259650642-11032-0/minicluster-data/master-0/wal/instance:
uuid: "65fef63633814a5aa8473856092cb9f2"
format_stamp: "Formatted at 2026-09-25 22:49:10 on dist-test-slave-5vdr"
server_key: "9d02f05c776139e8bd778f89cd3d0150"
server_key_iv: "17406a82f99661a14fa589597d75028c"
server_key_version: "encryptionkey@0"
I20260925 22:49:10.561964 597 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260925 22:49:10.565276 612 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:49:10.566695 597 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260925 22:49:10.566906 597 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790376259650642-11032-0/minicluster-data/master-0/data,/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790376259650642-11032-0/minicluster-data/master-0/wal
uuid: "65fef63633814a5aa8473856092cb9f2"
format_stamp: "Formatted at 2026-09-25 22:49:10 on dist-test-slave-5vdr"
server_key: "9d02f05c776139e8bd778f89cd3d0150"
server_key_iv: "17406a82f99661a14fa589597d75028c"
server_key_version: "encryptionkey@0"
I20260925 22:49:10.567066 597 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790376259650642-11032-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790376259650642-11032-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790376259650642-11032-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:49:10.591073 597 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:49:10.595131 597 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:49:10.595438 597 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:49:10.603952 597 rpc_server.cc:307] RPC server started. Bound to: 127.10.198.62:43075
I20260925 22:49:10.604050 664 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.198.62:43075 every 8 connection(s)
I20260925 22:49:10.605126 597 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790376259650642-11032-0/minicluster-data/master-0/data/info.pb
I20260925 22:49:10.608212 665 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:49:10.609640 11032 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrWXm8X/build/debug/bin/kudu as pid 597
I20260925 22:49:10.609824 11032 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790376259650642-11032-0/minicluster-data/master-0/wal/instance
I20260925 22:49:10.610118 11032 external_mini_cluster.cc:1428] Setting key b728da765d4b13c2975da5a3e7172b7a
I20260925 22:49:10.615010 665 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 65fef63633814a5aa8473856092cb9f2: Bootstrap starting.
I20260925 22:49:10.617892 665 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 65fef63633814a5aa8473856092cb9f2: Neither blocks nor log segments found. Creating new log.
I20260925 22:49:10.618801 665 log.cc:824] T 00000000000000000000000000000000 P 65fef63633814a5aa8473856092cb9f2: Log is configured to *not* fsync() on all Append() calls
I20260925 22:49:10.620358 668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:10.611642 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47177 (local address 127.10.198.62:43075)
0925 22:49:10.612253 (+ 611us) server_negotiation.cc:206] Beginning negotiation
0925 22:49:10.612272 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0925 22:49:10.612313 (+ 41us) server_negotiation.cc:407] Connection header received
0925 22:49:10.613087 (+ 774us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:49:10.613107 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:49:10.613492 (+ 385us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:49:10.613872 (+ 380us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:49:10.614309 (+ 437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:49:10.615397 (+ 1088us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:49:10.616475 (+ 1078us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:49:10.617024 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:49:10.617226 (+ 202us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:49:10.618186 (+ 960us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:49:10.618192 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:49:10.618208 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 22:49:10.618662 (+ 454us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:49:10.618730 (+ 68us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 22:49:10.618757 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:49:10.618895 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0925 22:49:10.619026 (+ 131us) server_negotiation.cc:299] Negotiation successful
0925 22:49:10.619283 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":425,"thread_start_us":188,"threads_started":1}
I20260925 22:49:10.621523 665 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 65fef63633814a5aa8473856092cb9f2: No bootstrap required, opened a new log
I20260925 22:49:10.624109 11032 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 22:49:10.626237 665 raft_consensus.cc:348] T 00000000000000000000000000000000 P 65fef63633814a5aa8473856092cb9f2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65fef63633814a5aa8473856092cb9f2" member_type: VOTER last_known_addr { host: "127.10.198.62" port: 43075 } }
I20260925 22:49:10.626695 665 raft_consensus.cc:374] T 00000000000000000000000000000000 P 65fef63633814a5aa8473856092cb9f2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:49:10.626806 665 raft_consensus.cc:729] T 00000000000000000000000000000000 P 65fef63633814a5aa8473856092cb9f2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 65fef63633814a5aa8473856092cb9f2, State: Initialized, Role: FOLLOWER
I20260925 22:49:10.627385 665 consensus_queue.cc:260] T 00000000000000000000000000000000 P 65fef63633814a5aa8473856092cb9f2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65fef63633814a5aa8473856092cb9f2" member_type: VOTER last_known_addr { host: "127.10.198.62" port: 43075 } }
I20260925 22:49:10.627622 665 raft_consensus.cc:388] T 00000000000000000000000000000000 P 65fef63633814a5aa8473856092cb9f2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 22:49:10.627705 665 raft_consensus.cc:482] T 00000000000000000000000000000000 P 65fef63633814a5aa8473856092cb9f2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 22:49:10.627825 665 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 65fef63633814a5aa8473856092cb9f2 [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:49:10.629213 665 raft_consensus.cc:504] T 00000000000000000000000000000000 P 65fef63633814a5aa8473856092cb9f2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65fef63633814a5aa8473856092cb9f2" member_type: VOTER last_known_addr { host: "127.10.198.62" port: 43075 } }
I20260925 22:49:10.629698 665 leader_election.cc:302] T 00000000000000000000000000000000 P 65fef63633814a5aa8473856092cb9f2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 65fef63633814a5aa8473856092cb9f2; no voters:
I20260925 22:49:10.630115 665 leader_election.cc:288] T 00000000000000000000000000000000 P 65fef63633814a5aa8473856092cb9f2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 22:49:10.630232 675 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 65fef63633814a5aa8473856092cb9f2 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 22:49:10.630509 675 raft_consensus.cc:686] T 00000000000000000000000000000000 P 65fef63633814a5aa8473856092cb9f2 [term 1 LEADER]: Becoming Leader. State: Replica: 65fef63633814a5aa8473856092cb9f2, State: Running, Role: LEADER
I20260925 22:49:10.630950 675 consensus_queue.cc:237] T 00000000000000000000000000000000 P 65fef63633814a5aa8473856092cb9f2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65fef63633814a5aa8473856092cb9f2" member_type: VOTER last_known_addr { host: "127.10.198.62" port: 43075 } }
I20260925 22:49:10.631893 668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:10.626468 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47881 (local address 127.10.198.62:43075)
0925 22:49:10.626641 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0925 22:49:10.626648 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 22:49:10.626777 (+ 129us) server_negotiation.cc:407] Connection header received
0925 22:49:10.626992 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:49:10.626996 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:49:10.627075 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:49:10.627167 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:49:10.627587 (+ 420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:49:10.628381 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:49:10.629596 (+ 1215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:49:10.629936 (+ 340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:49:10.630028 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:49:10.631013 (+ 985us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:49:10.631017 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:49:10.631020 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 22:49:10.631036 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:49:10.631069 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 22:49:10.631084 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:49:10.631151 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0925 22:49:10.631559 (+ 408us) server_negotiation.cc:299] Negotiation successful
0925 22:49:10.631623 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260925 22:49:10.631881 665 sys_catalog.cc:564] T 00000000000000000000000000000000 P 65fef63633814a5aa8473856092cb9f2 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 22:49:10.632992 677 sys_catalog.cc:455] T 00000000000000000000000000000000 P 65fef63633814a5aa8473856092cb9f2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 65fef63633814a5aa8473856092cb9f2. Latest consensus state: current_term: 1 leader_uuid: "65fef63633814a5aa8473856092cb9f2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65fef63633814a5aa8473856092cb9f2" member_type: VOTER last_known_addr { host: "127.10.198.62" port: 43075 } } }
I20260925 22:49:10.633157 677 sys_catalog.cc:458] T 00000000000000000000000000000000 P 65fef63633814a5aa8473856092cb9f2 [sys.catalog]: This master's current role is: LEADER
I20260925 22:49:10.633517 676 sys_catalog.cc:455] T 00000000000000000000000000000000 P 65fef63633814a5aa8473856092cb9f2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "65fef63633814a5aa8473856092cb9f2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65fef63633814a5aa8473856092cb9f2" member_type: VOTER last_known_addr { host: "127.10.198.62" port: 43075 } } }
I20260925 22:49:10.633620 676 sys_catalog.cc:458] T 00000000000000000000000000000000 P 65fef63633814a5aa8473856092cb9f2 [sys.catalog]: This master's current role is: LEADER
I20260925 22:49:10.633903 684 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 22:49:10.637130 684 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 22:49:10.643803 684 catalog_manager.cc:1347] Generated new cluster ID: 464721e36a354ad9bc695a66860339de
I20260925 22:49:10.644104 684 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 22:49:10.654299 684 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 22:49:10.655591 684 catalog_manager.cc:1504] Loading token signing keys...
I20260925 22:49:10.661698 684 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 65fef63633814a5aa8473856092cb9f2: Generated new TSK 0
I20260925 22:49:10.662729 684 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 22:49:10.675479 11032 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrWXm8X/build/debug/bin/kudu with pid 597
2026-09-25T22:49:10Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1517 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790376259650642-11032-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 22:49:10 dist-test-slave-5vdr krb5kdc[696](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 22:49:10 dist-test-slave-5vdr krb5kdc[696](info): set up 2 sockets
Sep 25 22:49:10 dist-test-slave-5vdr krb5kdc[696](info): commencing operation
krb5kdc: starting...
W20260925 22:49:11.720350 11032 mini_kdc.cc:121] Time spent starting KDC: real 1.024s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790376259650642-11032-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790376259650642-11032-0/krb5kdc/test-user.keytab.
Sep 25 22:49:11 dist-test-slave-5vdr krb5kdc[696](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376551, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T22:49:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T22:49:11Z Disabled control of system clock
WARNING: no policy specified for kudu/127.10.198.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.198.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.198.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790376259650642-11032-0/krb5kdc/kudu_127.10.198.62.keytab.
Entry for principal kudu/127.10.198.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790376259650642-11032-0/krb5kdc/kudu_127.10.198.62.keytab.
I20260925 22:49:11.851608 11032 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrWXm8X/build/debug/bin/kudu
/tmp/dist-test-taskrWXm8X/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790376259650642-11032-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790376259650642-11032-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790376259650642-11032-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790376259650642-11032-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.198.62:33645
--webserver_interface=127.10.198.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790376259650642-11032-0/krb5kdc/kudu_127.10.198.62.keytab
--principal=kudu/127.10.198.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.198.20:35765
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.198.62:33645
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790376259650642-11032-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790376259650642-11032-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790376259650642-11032-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:49:11.977697 709 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:49:11.978113 709 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:49:11.978166 709 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:49:11.982528 709 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 22:49:11.982620 709 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:49:11.982640 709 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:49:11.982658 709 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 22:49:11.982673 709 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 22:49:11.988739 709 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.198.20:35765
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790376259650642-11032-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790376259650642-11032-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.198.62:33645
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790376259650642-11032-0/krb5kdc/kudu_127.10.198.62.keytab
--principal=kudu/127.10.198.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.198.62:33645
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790376259650642-11032-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.198.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.709
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790376259650642-11032-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:09:16 UTC on ccycloud.quasar-afmhyw.root.comops.site
build id 4673379
I20260925 22:49:11.989956 709 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:49:11.991078 709 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:49:12.000620 717 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:49:12.000623 715 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:49:12.000623 714 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:49:12.001019 709 server_base.cc:1034] running on GCE node
I20260925 22:49:12.001796 709 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:49:12.003187 709 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:49:12.004422 709 hybrid_clock.cc:648] HybridClock initialized: now 1790376552004401 us; error 48 us; skew 500 ppm
Sep 25 22:49:12 dist-test-slave-5vdr krb5kdc[696](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376552, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.198.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:49:12.007622 709 init.cc:377] Logged in from keytab as kudu/127.10.198.62@KRBTEST.COM (short username kudu)
I20260925 22:49:12.008879 709 webserver.cc:460] Webserver started at http://127.10.198.62:40531/ using document root <none> and password file <none>
I20260925 22:49:12.009536 709 fs_manager.cc:359] Metadata directory not provided
I20260925 22:49:12.009615 709 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:49:12.009881 709 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:49:12.012485 709 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790376259650642-11032-0/minicluster-data/master-0/data/instance:
uuid: "2174a6ef264b41b9836e6bb3eb51ad17"
format_stamp: "Formatted at 2026-09-25 22:49:12 on dist-test-slave-5vdr"
server_key: "99fe015829e02e9edb13a54178584959"
server_key_iv: "763708fa78caf944fe2b720c0fad65db"
server_key_version: "encryptionkey@0"
I20260925 22:49:12.013237 709 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790376259650642-11032-0/minicluster-data/master-0/wal/instance:
uuid: "2174a6ef264b41b9836e6bb3eb51ad17"
format_stamp: "Formatted at 2026-09-25 22:49:12 on dist-test-slave-5vdr"
server_key: "99fe015829e02e9edb13a54178584959"
server_key_iv: "763708fa78caf944fe2b720c0fad65db"
server_key_version: "encryptionkey@0"
I20260925 22:49:12.018066 709 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260925 22:49:12.021719 724 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:49:12.023432 709 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260925 22:49:12.023670 709 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790376259650642-11032-0/minicluster-data/master-0/data,/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790376259650642-11032-0/minicluster-data/master-0/wal
uuid: "2174a6ef264b41b9836e6bb3eb51ad17"
format_stamp: "Formatted at 2026-09-25 22:49:12 on dist-test-slave-5vdr"
server_key: "99fe015829e02e9edb13a54178584959"
server_key_iv: "763708fa78caf944fe2b720c0fad65db"
server_key_version: "encryptionkey@0"
I20260925 22:49:12.023916 709 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790376259650642-11032-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790376259650642-11032-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790376259650642-11032-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:49:12.029345 709 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:49:12.033159 709 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:49:12.033620 709 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:49:12.042429 709 rpc_server.cc:307] RPC server started. Bound to: 127.10.198.62:33645
I20260925 22:49:12.042450 776 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.198.62:33645 every 8 connection(s)
I20260925 22:49:12.044124 709 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790376259650642-11032-0/minicluster-data/master-0/data/info.pb
I20260925 22:49:12.048677 777 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:49:12.049611 11032 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrWXm8X/build/debug/bin/kudu as pid 709
I20260925 22:49:12.049793 11032 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790376259650642-11032-0/minicluster-data/master-0/wal/instance
I20260925 22:49:12.050118 11032 external_mini_cluster.cc:1428] Setting key b3d42b7203ca04b4f1398f6b52726373
I20260925 22:49:12.056313 780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:12.051833 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57799 (local address 127.10.198.62:33645)
0925 22:49:12.052408 (+ 575us) server_negotiation.cc:206] Beginning negotiation
0925 22:49:12.052425 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0925 22:49:12.052467 (+ 42us) server_negotiation.cc:407] Connection header received
0925 22:49:12.053196 (+ 729us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:49:12.053217 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:49:12.053564 (+ 347us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:49:12.053982 (+ 418us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:49:12.054906 (+ 924us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:49:12.054911 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:49:12.054925 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 22:49:12.054937 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:49:12.054999 (+ 62us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 22:49:12.055020 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:49:12.055191 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0925 22:49:12.055270 (+ 79us) server_negotiation.cc:299] Negotiation successful
0925 22:49:12.055450 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":378,"thread_start_us":163,"threads_started":1}
I20260925 22:49:12.056382 777 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2174a6ef264b41b9836e6bb3eb51ad17: Bootstrap starting.
I20260925 22:49:12.059373 777 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2174a6ef264b41b9836e6bb3eb51ad17: Neither blocks nor log segments found. Creating new log.
I20260925 22:49:12.060302 11032 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 22:49:12.060513 777 log.cc:824] T 00000000000000000000000000000000 P 2174a6ef264b41b9836e6bb3eb51ad17: Log is configured to *not* fsync() on all Append() calls
I20260925 22:49:12.063091 777 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2174a6ef264b41b9836e6bb3eb51ad17: No bootstrap required, opened a new log
I20260925 22:49:12.064721 780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:12.062614 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40431 (local address 127.10.198.62:33645)
0925 22:49:12.062772 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0925 22:49:12.062777 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:49:12.062920 (+ 143us) server_negotiation.cc:407] Connection header received
0925 22:49:12.063015 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:49:12.063018 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:49:12.063057 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:49:12.063164 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:49:12.064185 (+ 1021us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:49:12.064189 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:49:12.064192 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 22:49:12.064207 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:49:12.064243 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 22:49:12.064258 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:49:12.064367 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0925 22:49:12.064536 (+ 169us) server_negotiation.cc:299] Negotiation successful
0925 22:49:12.064582 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260925 22:49:12.067782 777 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2174a6ef264b41b9836e6bb3eb51ad17 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2174a6ef264b41b9836e6bb3eb51ad17" member_type: VOTER last_known_addr { host: "127.10.198.62" port: 33645 } }
I20260925 22:49:12.068200 777 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2174a6ef264b41b9836e6bb3eb51ad17 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 22:49:12.068279 777 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2174a6ef264b41b9836e6bb3eb51ad17 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2174a6ef264b41b9836e6bb3eb51ad17, State: Initialized, Role: FOLLOWER
I20260925 22:49:12.068818 777 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2174a6ef264b41b9836e6bb3eb51ad17 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2174a6ef264b41b9836e6bb3eb51ad17" member_type: VOTER last_known_addr { host: "127.10.198.62" port: 33645 } }
I20260925 22:49:12.068997 777 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2174a6ef264b41b9836e6bb3eb51ad17 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 22:49:12.069072 777 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2174a6ef264b41b9836e6bb3eb51ad17 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 22:49:12.069182 777 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2174a6ef264b41b9836e6bb3eb51ad17 [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:49:12.070286 777 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2174a6ef264b41b9836e6bb3eb51ad17 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2174a6ef264b41b9836e6bb3eb51ad17" member_type: VOTER last_known_addr { host: "127.10.198.62" port: 33645 } }
I20260925 22:49:12.070715 777 leader_election.cc:302] T 00000000000000000000000000000000 P 2174a6ef264b41b9836e6bb3eb51ad17 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2174a6ef264b41b9836e6bb3eb51ad17; no voters:
I20260925 22:49:12.071134 777 leader_election.cc:288] T 00000000000000000000000000000000 P 2174a6ef264b41b9836e6bb3eb51ad17 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 22:49:12.071297 787 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2174a6ef264b41b9836e6bb3eb51ad17 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 22:49:12.071561 787 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2174a6ef264b41b9836e6bb3eb51ad17 [term 1 LEADER]: Becoming Leader. State: Replica: 2174a6ef264b41b9836e6bb3eb51ad17, State: Running, Role: LEADER
I20260925 22:49:12.072041 787 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2174a6ef264b41b9836e6bb3eb51ad17 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2174a6ef264b41b9836e6bb3eb51ad17" member_type: VOTER last_known_addr { host: "127.10.198.62" port: 33645 } }
I20260925 22:49:12.072618 777 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2174a6ef264b41b9836e6bb3eb51ad17 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 22:49:12.074244 789 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2174a6ef264b41b9836e6bb3eb51ad17 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2174a6ef264b41b9836e6bb3eb51ad17. Latest consensus state: current_term: 1 leader_uuid: "2174a6ef264b41b9836e6bb3eb51ad17" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2174a6ef264b41b9836e6bb3eb51ad17" member_type: VOTER last_known_addr { host: "127.10.198.62" port: 33645 } } }
I20260925 22:49:12.074728 789 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2174a6ef264b41b9836e6bb3eb51ad17 [sys.catalog]: This master's current role is: LEADER
I20260925 22:49:12.075439 788 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2174a6ef264b41b9836e6bb3eb51ad17 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2174a6ef264b41b9836e6bb3eb51ad17" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2174a6ef264b41b9836e6bb3eb51ad17" member_type: VOTER last_known_addr { host: "127.10.198.62" port: 33645 } } }
I20260925 22:49:12.075659 788 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2174a6ef264b41b9836e6bb3eb51ad17 [sys.catalog]: This master's current role is: LEADER
I20260925 22:49:12.075620 798 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 22:49:12.079594 798 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 22:49:12.087473 798 catalog_manager.cc:1347] Generated new cluster ID: 5fb2dafe2d1b4d879e39e4d6000f94e9
I20260925 22:49:12.087580 798 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 22:49:12.103060 798 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 22:49:12.103494 798 catalog_manager.cc:1504] Loading token signing keys...
I20260925 22:49:12.113190 798 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2174a6ef264b41b9836e6bb3eb51ad17: Generated new TSK 0
I20260925 22:49:12.114040 798 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 22:49:12.121769 11032 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrWXm8X/build/debug/bin/kudu with pid 709
2026-09-25T22:49:12Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1446 ms)
[----------] 18 tests from AuthTokenIssuingTest (26476 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790376259650642-11032-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 22:49:12 dist-test-slave-5vdr krb5kdc[808](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 22:49:12 dist-test-slave-5vdr krb5kdc[808](info): set up 2 sockets
Sep 25 22:49:12 dist-test-slave-5vdr krb5kdc[808](info): commencing operation
krb5kdc: starting...
W20260925 22:49:13.208806 11032 mini_kdc.cc:121] Time spent starting KDC: real 1.066s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790376259650642-11032-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790376259650642-11032-0/krb5kdc/test-user.keytab.
Sep 25 22:49:13 dist-test-slave-5vdr krb5kdc[808](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376553, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T22:49:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T22:49:13Z Disabled control of system clock
WARNING: no policy specified for kudu/127.10.198.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.198.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.198.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790376259650642-11032-0/krb5kdc/kudu_127.10.198.62.keytab.
Entry for principal kudu/127.10.198.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790376259650642-11032-0/krb5kdc/kudu_127.10.198.62.keytab.
I20260925 22:49:13.340191 11032 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrWXm8X/build/debug/bin/kudu
/tmp/dist-test-taskrWXm8X/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790376259650642-11032-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790376259650642-11032-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790376259650642-11032-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790376259650642-11032-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.198.62:46393
--webserver_interface=127.10.198.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790376259650642-11032-0/krb5kdc/kudu_127.10.198.62.keytab
--principal=kudu/127.10.198.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.198.20:33069
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.198.62:46393,127.10.198.61:46873,127.10.198.60:38695
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790376259650642-11032-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790376259650642-11032-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790376259650642-11032-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:49:13.460731 821 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260925 22:49:13.461169 821 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:49:13.461244 821 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:49:13.461323 821 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:49:13.465770 821 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 22:49:13.465909 821 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:49:13.465946 821 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:49:13.465987 821 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 22:49:13.466027 821 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 22:49:13.472010 821 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.198.20:33069
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790376259650642-11032-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790376259650642-11032-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.198.62:46393,127.10.198.61:46873,127.10.198.60:38695
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790376259650642-11032-0/krb5kdc/kudu_127.10.198.62.keytab
--principal=kudu/127.10.198.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.198.62:46393
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790376259650642-11032-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.198.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.821
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790376259650642-11032-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:09:16 UTC on ccycloud.quasar-afmhyw.root.comops.site
build id 4673379
I20260925 22:49:13.473308 821 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:49:13.474449 821 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:49:13.484185 826 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:49:13.484200 829 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:49:13.484283 821 server_base.cc:1034] running on GCE node
W20260925 22:49:13.484200 827 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:49:13.485152 821 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:49:13.487387 821 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:49:13.488655 821 hybrid_clock.cc:648] HybridClock initialized: now 1790376553488631 us; error 58 us; skew 500 ppm
Sep 25 22:49:13 dist-test-slave-5vdr krb5kdc[808](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376553, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.198.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:49:13.492070 821 init.cc:377] Logged in from keytab as kudu/127.10.198.62@KRBTEST.COM (short username kudu)
I20260925 22:49:13.493323 821 webserver.cc:460] Webserver started at http://127.10.198.62:39587/ using document root <none> and password file <none>
I20260925 22:49:13.494438 821 fs_manager.cc:359] Metadata directory not provided
I20260925 22:49:13.494735 821 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:49:13.495110 821 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:49:13.497978 821 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790376259650642-11032-0/minicluster-data/master-0/data/instance:
uuid: "2f6b93a17812478199d694a247f1af35"
format_stamp: "Formatted at 2026-09-25 22:49:13 on dist-test-slave-5vdr"
server_key: "fc02b4308db3dd0367754cefbcbc2565"
server_key_iv: "391b53107738803294af62c10d3f2db2"
server_key_version: "encryptionkey@0"
I20260925 22:49:13.498690 821 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790376259650642-11032-0/minicluster-data/master-0/wal/instance:
uuid: "2f6b93a17812478199d694a247f1af35"
format_stamp: "Formatted at 2026-09-25 22:49:13 on dist-test-slave-5vdr"
server_key: "fc02b4308db3dd0367754cefbcbc2565"
server_key_iv: "391b53107738803294af62c10d3f2db2"
server_key_version: "encryptionkey@0"
I20260925 22:49:13.503523 821 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.005s
I20260925 22:49:13.506580 836 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:49:13.508253 821 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 22:49:13.508445 821 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790376259650642-11032-0/minicluster-data/master-0/data,/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790376259650642-11032-0/minicluster-data/master-0/wal
uuid: "2f6b93a17812478199d694a247f1af35"
format_stamp: "Formatted at 2026-09-25 22:49:13 on dist-test-slave-5vdr"
server_key: "fc02b4308db3dd0367754cefbcbc2565"
server_key_iv: "391b53107738803294af62c10d3f2db2"
server_key_version: "encryptionkey@0"
I20260925 22:49:13.508605 821 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790376259650642-11032-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790376259650642-11032-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790376259650642-11032-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:49:13.524184 821 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:49:13.527704 821 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:49:13.528008 821 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:49:13.537276 821 rpc_server.cc:307] RPC server started. Bound to: 127.10.198.62:46393
I20260925 22:49:13.537642 888 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.198.62:46393 every 8 connection(s)
I20260925 22:49:13.538408 821 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790376259650642-11032-0/minicluster-data/master-0/data/info.pb
I20260925 22:49:13.538695 11032 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrWXm8X/build/debug/bin/kudu as pid 821
I20260925 22:49:13.538852 11032 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790376259650642-11032-0/minicluster-data/master-0/wal/instance
I20260925 22:49:13.539179 11032 external_mini_cluster.cc:1428] Setting key d6289e1aa799f7294d5f66c596960f4f
I20260925 22:49:13.542120 889 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:49:13.546525 889 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.198.62" port: 46393 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.10.198.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.198.61@KRBTEST.COM" created.
Sep 25 22:49:13 dist-test-slave-5vdr krb5kdc[808](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376553, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.198.62@KRBTEST.COM for kudu/127.10.198.62@KRBTEST.COM
I20260925 22:49:13.567116 892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:13.549345 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.198.62:46393 (local address 127.0.0.1:41838)
0925 22:49:13.550097 (+ 752us) negotiation.cc:107] Waiting for socket to connect
0925 22:49:13.550152 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0925 22:49:13.551398 (+ 1246us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:49:13.552446 (+ 1048us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:49:13.552455 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:49:13.553041 (+ 586us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:49:13.553497 (+ 456us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:49:13.553524 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:49:13.554864 (+ 1340us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:49:13.554867 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:49:13.556786 (+ 1919us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:49:13.556807 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:49:13.557675 (+ 868us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:49:13.557682 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:49:13.557806 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:49:13.558787 (+ 981us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:49:13.558814 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:49:13.560965 (+ 2151us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:49:13.563691 (+ 2726us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:49:13.563705 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:49:13.563724 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:49:13.564068 (+ 344us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:49:13.564500 (+ 432us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:49:13.564503 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:49:13.564504 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:49:13.564635 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:49:13.565427 (+ 792us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:49:13.565470 (+ 43us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:49:13.565648 (+ 178us) client_negotiation.cc:769] Sending connection context
0925 22:49:13.565809 (+ 161us) client_negotiation.cc:240] Negotiation successful
0925 22:49:13.566173 (+ 364us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":523,"thread_start_us":221,"threads_started":1}
I20260925 22:49:13.567132 893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:13.550553 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41838 (local address 127.10.198.62:46393)
0925 22:49:13.550872 (+ 319us) server_negotiation.cc:206] Beginning negotiation
0925 22:49:13.550888 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0925 22:49:13.550937 (+ 49us) server_negotiation.cc:407] Connection header received
0925 22:49:13.551785 (+ 848us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:49:13.551795 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:49:13.552121 (+ 326us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:49:13.552380 (+ 259us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:49:13.553720 (+ 1340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:49:13.554685 (+ 965us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:49:13.556995 (+ 2310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:49:13.557457 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:49:13.557607 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:49:13.561175 (+ 3568us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:49:13.561182 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:49:13.561202 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:49:13.561228 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:49:13.563473 (+ 2245us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:49:13.564214 (+ 741us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:49:13.564222 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:49:13.564228 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:49:13.564353 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:49:13.564735 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:49:13.564738 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:49:13.564758 (+ 20us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:49:13.565293 (+ 535us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:49:13.565450 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0925 22:49:13.565968 (+ 518us) server_negotiation.cc:299] Negotiation successful
0925 22:49:13.566288 (+ 320us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":105,"server-negotiator.queue_time_us":215,"thread_start_us":130,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.198.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790376259650642-11032-0/krb5kdc/kudu_127.10.198.61.keytab.
Entry for principal kudu/127.10.198.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790376259650642-11032-0/krb5kdc/kudu_127.10.198.61.keytab.
I20260925 22:49:13.570633 889 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.198.61" port: 46873 } has no permanent_uuid. Determining permanent_uuid...
I20260925 22:49:13.571236 11032 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrWXm8X/build/debug/bin/kudu
/tmp/dist-test-taskrWXm8X/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790376259650642-11032-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790376259650642-11032-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790376259650642-11032-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790376259650642-11032-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.198.61:46873
--webserver_interface=127.10.198.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790376259650642-11032-0/krb5kdc/kudu_127.10.198.61.keytab
--principal=kudu/127.10.198.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.198.20:33069
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.198.62:46393,127.10.198.61:46873,127.10.198.60:38695
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790376259650642-11032-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790376259650642-11032-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790376259650642-11032-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 22:49:13.571575 892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:13.571240 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.198.61:46873 (local address 127.0.0.1:54328)
0925 22:49:13.571399 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0925 22:49:13.571495 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.198.61:46873: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
W20260925 22:49:13.571825 837 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.10.198.61:46873: connect: Connection refused (error 111)
I20260925 22:49:13.572877 892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:13.572622 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.198.61:46873 (local address 127.0.0.1:54334)
0925 22:49:13.572745 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0925 22:49:13.572804 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.198.61:46873: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260925 22:49:13.573175 889 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.10.198.61:46873: Network error: Client connection negotiation failed: client connection to 127.10.198.61:46873: connect: Connection refused (error 111)
I20260925 22:49:13.622478 889 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.198.61" port: 46873 } attempt: 1
I20260925 22:49:13.623632 892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:13.623234 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.198.61:46873 (local address 127.0.0.1:54348)
0925 22:49:13.623462 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0925 22:49:13.623538 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.198.61:46873: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104}
I20260925 22:49:13.624696 892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:13.624422 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.198.61:46873 (local address 127.0.0.1:54362)
0925 22:49:13.624561 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0925 22:49:13.624622 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.198.61:46873: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
W20260925 22:49:13.624866 889 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.10.198.61:46873: Network error: Client connection negotiation failed: client connection to 127.10.198.61:46873: connect: Connection refused (error 111)
I20260925 22:49:13.693069 889 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.198.61" port: 46873 } attempt: 2
I20260925 22:49:13.694176 892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:13.693844 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.198.61:46873 (local address 127.0.0.1:54374)
0925 22:49:13.694009 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0925 22:49:13.694082 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.198.61:46873: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260925 22:49:13.695037 892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:13.694771 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.198.61:46873 (local address 127.0.0.1:54388)
0925 22:49:13.694908 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0925 22:49:13.694966 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.198.61:46873: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260925 22:49:13.695313 889 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.10.198.61:46873: Network error: Client connection negotiation failed: client connection to 127.10.198.61:46873: connect: Connection refused (error 111)
W20260925 22:49:13.702967 896 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260925 22:49:13.703318 896 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:49:13.703343 896 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:49:13.703387 896 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:49:13.707779 896 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 22:49:13.707957 896 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:49:13.707985 896 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:49:13.708002 896 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 22:49:13.708019 896 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 22:49:13.714577 896 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.198.20:33069
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790376259650642-11032-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790376259650642-11032-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.10.198.62:46393,127.10.198.61:46873,127.10.198.60:38695
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790376259650642-11032-0/krb5kdc/kudu_127.10.198.61.keytab
--principal=kudu/127.10.198.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.198.61:46873
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790376259650642-11032-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.10.198.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.896
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790376259650642-11032-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:09:16 UTC on ccycloud.quasar-afmhyw.root.comops.site
build id 4673379
I20260925 22:49:13.715739 896 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:49:13.716953 896 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:49:13.727228 904 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:49:13.727246 901 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:49:13.727624 896 server_base.cc:1034] running on GCE node
W20260925 22:49:13.727249 902 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:49:13.728648 896 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:49:13.730093 896 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:49:13.731359 896 hybrid_clock.cc:648] HybridClock initialized: now 1790376553731325 us; error 54 us; skew 500 ppm
Sep 25 22:49:13 dist-test-slave-5vdr krb5kdc[808](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376553, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.198.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:49:13.734673 896 init.cc:377] Logged in from keytab as kudu/127.10.198.61@KRBTEST.COM (short username kudu)
I20260925 22:49:13.735818 896 webserver.cc:460] Webserver started at http://127.10.198.61:43283/ using document root <none> and password file <none>
I20260925 22:49:13.736517 896 fs_manager.cc:359] Metadata directory not provided
I20260925 22:49:13.736586 896 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:49:13.736778 896 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:49:13.739075 896 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790376259650642-11032-0/minicluster-data/master-1/data/instance:
uuid: "f3fc2bc68b2f4752a289c342d2a82bad"
format_stamp: "Formatted at 2026-09-25 22:49:13 on dist-test-slave-5vdr"
server_key: "3359c0d2886a19ebe0277739835386aa"
server_key_iv: "a1582e38db5a85b42b592eff3f91a2f7"
server_key_version: "encryptionkey@0"
I20260925 22:49:13.739687 896 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790376259650642-11032-0/minicluster-data/master-1/wal/instance:
uuid: "f3fc2bc68b2f4752a289c342d2a82bad"
format_stamp: "Formatted at 2026-09-25 22:49:13 on dist-test-slave-5vdr"
server_key: "3359c0d2886a19ebe0277739835386aa"
server_key_iv: "a1582e38db5a85b42b592eff3f91a2f7"
server_key_version: "encryptionkey@0"
I20260925 22:49:13.745283 896 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.003s
I20260925 22:49:13.749400 911 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:49:13.750948 896 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.005s sys 0.000s
I20260925 22:49:13.751114 896 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790376259650642-11032-0/minicluster-data/master-1/data,/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790376259650642-11032-0/minicluster-data/master-1/wal
uuid: "f3fc2bc68b2f4752a289c342d2a82bad"
format_stamp: "Formatted at 2026-09-25 22:49:13 on dist-test-slave-5vdr"
server_key: "3359c0d2886a19ebe0277739835386aa"
server_key_iv: "a1582e38db5a85b42b592eff3f91a2f7"
server_key_version: "encryptionkey@0"
I20260925 22:49:13.751258 896 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790376259650642-11032-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790376259650642-11032-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790376259650642-11032-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:49:13.772087 896 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:49:13.776033 896 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:49:13.776232 896 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:49:13.785180 896 rpc_server.cc:307] RPC server started. Bound to: 127.10.198.61:46873
I20260925 22:49:13.785583 963 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.198.61:46873 every 8 connection(s)
I20260925 22:49:13.786504 889 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.198.61" port: 46873 } attempt: 3
I20260925 22:49:13.786646 896 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790376259650642-11032-0/minicluster-data/master-1/data/info.pb
I20260925 22:49:13.789929 964 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:49:13.790004 11032 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrWXm8X/build/debug/bin/kudu as pid 896
I20260925 22:49:13.790150 11032 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790376259650642-11032-0/minicluster-data/master-1/wal/instance
I20260925 22:49:13.790521 11032 external_mini_cluster.cc:1428] Setting key 1973eaf8a24033c1ca0d5d13a979ac80
I20260925 22:49:13.795703 964 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.198.62" port: 46393 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 22:49:13 dist-test-slave-5vdr krb5kdc[808](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376553, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.198.62@KRBTEST.COM for kudu/127.10.198.61@KRBTEST.COM
WARNING: no policy specified for kudu/127.10.198.60@KRBTEST.COM; defaulting to no policy
I20260925 22:49:13.803262 892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:13.787090 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.198.61:46873 (local address 127.0.0.1:54402)
0925 22:49:13.787273 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0925 22:49:13.787294 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0925 22:49:13.787407 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:49:13.790300 (+ 2893us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:49:13.790304 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:49:13.790545 (+ 241us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:49:13.790729 (+ 184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:49:13.790740 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:49:13.793280 (+ 2540us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:49:13.793285 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:49:13.794048 (+ 763us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:49:13.794068 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:49:13.795207 (+ 1139us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:49:13.795214 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:49:13.795368 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:49:13.796093 (+ 725us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:49:13.796119 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:49:13.797895 (+ 1776us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:49:13.800854 (+ 2959us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:49:13.800861 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:49:13.800866 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:49:13.801125 (+ 259us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:49:13.801617 (+ 492us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:49:13.801620 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:49:13.801622 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:49:13.801732 (+ 110us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:49:13.802601 (+ 869us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:49:13.802606 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:49:13.802766 (+ 160us) client_negotiation.cc:769] Sending connection context
0925 22:49:13.802889 (+ 123us) client_negotiation.cc:240] Negotiation successful
0925 22:49:13.803038 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":85}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.198.60@KRBTEST.COM" created.
I20260925 22:49:13.804965 966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:13.788019 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54402 (local address 127.10.198.61:46873)
0925 22:49:13.788509 (+ 490us) server_negotiation.cc:206] Beginning negotiation
0925 22:49:13.788532 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0925 22:49:13.788584 (+ 52us) server_negotiation.cc:407] Connection header received
0925 22:49:13.789327 (+ 743us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:49:13.789347 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:49:13.789827 (+ 480us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:49:13.791019 (+ 1192us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:49:13.791566 (+ 547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:49:13.793064 (+ 1498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:49:13.794247 (+ 1183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:49:13.795006 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:49:13.795200 (+ 194us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:49:13.798144 (+ 2944us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:49:13.798151 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:49:13.798172 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:49:13.798217 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:49:13.800686 (+ 2469us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:49:13.801259 (+ 573us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:49:13.801266 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:49:13.801274 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:49:13.801477 (+ 203us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:49:13.801837 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:49:13.801840 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:49:13.801841 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:49:13.802305 (+ 464us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:49:13.802490 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0925 22:49:13.803286 (+ 796us) server_negotiation.cc:299] Negotiation successful
0925 22:49:13.803680 (+ 394us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":325,"thread_start_us":176,"threads_started":1}
Sep 25 22:49:13 dist-test-slave-5vdr krb5kdc[808](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376553, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.198.61@KRBTEST.COM for kudu/127.10.198.62@KRBTEST.COM
I20260925 22:49:13.807448 889 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.198.60" port: 38695 } has no permanent_uuid. Determining permanent_uuid...
I20260925 22:49:13.808379 892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:13.808061 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.198.60:38695 (local address 127.0.0.1:49236)
0925 22:49:13.808206 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0925 22:49:13.808281 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.198.60:38695: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20260925 22:49:13.809566 892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:13.809262 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.198.60:38695 (local address 127.0.0.1:49240)
0925 22:49:13.809442 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0925 22:49:13.809500 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.198.60:38695: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103}
W20260925 22:49:13.809759 889 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.10.198.60:38695: Network error: Client connection negotiation failed: client connection to 127.10.198.60:38695: connect: Connection refused (error 111)
I20260925 22:49:13.812153 968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:13.798054 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.198.62:46393 (local address 127.0.0.1:41854)
0925 22:49:13.798589 (+ 535us) negotiation.cc:107] Waiting for socket to connect
0925 22:49:13.798662 (+ 73us) client_negotiation.cc:174] Beginning negotiation
0925 22:49:13.798888 (+ 226us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:49:13.799261 (+ 373us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:49:13.799269 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:49:13.799929 (+ 660us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:49:13.800157 (+ 228us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:49:13.800171 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:49:13.801420 (+ 1249us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:49:13.801423 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:49:13.802633 (+ 1210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:49:13.802647 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:49:13.803662 (+ 1015us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:49:13.803666 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:49:13.803785 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:49:13.805425 (+ 1640us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:49:13.805439 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:49:13.807557 (+ 2118us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:49:13.810039 (+ 2482us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:49:13.810052 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:49:13.810058 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:49:13.810377 (+ 319us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:49:13.810767 (+ 390us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:49:13.810769 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:49:13.810771 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:49:13.810854 (+ 83us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:49:13.811436 (+ 582us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:49:13.811443 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:49:13.811572 (+ 129us) client_negotiation.cc:769] Sending connection context
0925 22:49:13.811706 (+ 134us) client_negotiation.cc:240] Negotiation successful
0925 22:49:13.811981 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":306,"thread_start_us":154,"threads_started":1}
I20260925 22:49:13.812973 893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:13.798086 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41854 (local address 127.10.198.62:46393)
0925 22:49:13.798242 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0925 22:49:13.798250 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 22:49:13.798772 (+ 522us) server_negotiation.cc:407] Connection header received
0925 22:49:13.799017 (+ 245us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:49:13.799022 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:49:13.799084 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:49:13.799189 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:49:13.800566 (+ 1377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:49:13.801262 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:49:13.802825 (+ 1563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:49:13.803394 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:49:13.803529 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:49:13.807720 (+ 4191us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:49:13.807725 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:49:13.807728 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:49:13.807757 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:49:13.809887 (+ 2130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:49:13.810513 (+ 626us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:49:13.810521 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:49:13.810524 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:49:13.810645 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:49:13.810958 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:49:13.810966 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:49:13.810969 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:49:13.811290 (+ 321us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:49:13.811433 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0925 22:49:13.812559 (+ 1126us) server_negotiation.cc:299] Negotiation successful
0925 22:49:13.812758 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":64}
I20260925 22:49:13.814272 964 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.198.61" port: 46873 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 22:49:13 dist-test-slave-5vdr krb5kdc[808](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376553, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.198.61@KRBTEST.COM for kudu/127.10.198.61@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.198.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790376259650642-11032-0/krb5kdc/kudu_127.10.198.60.keytab.
Entry for principal kudu/127.10.198.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790376259650642-11032-0/krb5kdc/kudu_127.10.198.60.keytab.
I20260925 22:49:13.824290 11032 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrWXm8X/build/debug/bin/kudu
/tmp/dist-test-taskrWXm8X/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790376259650642-11032-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790376259650642-11032-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790376259650642-11032-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790376259650642-11032-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.198.60:38695
--webserver_interface=127.10.198.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790376259650642-11032-0/krb5kdc/kudu_127.10.198.60.keytab
--principal=kudu/127.10.198.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.198.20:33069
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.198.62:46393,127.10.198.61:46873,127.10.198.60:38695
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790376259650642-11032-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790376259650642-11032-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790376259650642-11032-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 22:49:13.824518 968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:13.814843 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.198.61:46873 (local address 127.0.0.1:54410)
0925 22:49:13.814976 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0925 22:49:13.814991 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 22:49:13.815078 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:49:13.815345 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:49:13.815348 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:49:13.815540 (+ 192us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:49:13.815631 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:49:13.815637 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:49:13.816551 (+ 914us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:49:13.816558 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:49:13.817082 (+ 524us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:49:13.817090 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:49:13.817583 (+ 493us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:49:13.817586 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:49:13.817677 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:49:13.818238 (+ 561us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:49:13.818257 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:49:13.820025 (+ 1768us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:49:13.822276 (+ 2251us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:49:13.822281 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:49:13.822283 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:49:13.822512 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:49:13.822857 (+ 345us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:49:13.822860 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:49:13.822862 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:49:13.822952 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:49:13.823751 (+ 799us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:49:13.823754 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:49:13.823918 (+ 164us) client_negotiation.cc:769] Sending connection context
0925 22:49:13.824053 (+ 135us) client_negotiation.cc:240] Negotiation successful
0925 22:49:13.824309 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":50,"mutex_wait_us":25}
I20260925 22:49:13.824518 966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:13.814993 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54410 (local address 127.10.198.61:46873)
0925 22:49:13.815078 (+ 85us) server_negotiation.cc:206] Beginning negotiation
0925 22:49:13.815083 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:49:13.815095 (+ 12us) server_negotiation.cc:407] Connection header received
0925 22:49:13.815207 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:49:13.815209 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:49:13.815244 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:49:13.815330 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:49:13.815747 (+ 417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:49:13.816438 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:49:13.817223 (+ 785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:49:13.817454 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:49:13.817577 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:49:13.820187 (+ 2610us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:49:13.820192 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:49:13.820196 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:49:13.820231 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:49:13.822076 (+ 1845us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:49:13.822615 (+ 539us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:49:13.822622 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:49:13.822624 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:49:13.822702 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:49:13.823045 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:49:13.823050 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:49:13.823052 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:49:13.823577 (+ 525us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:49:13.823764 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0925 22:49:13.824099 (+ 335us) server_negotiation.cc:299] Negotiation successful
0925 22:49:13.824367 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":140,"server-negotiator.queue_time_us":30}
I20260925 22:49:13.825915 964 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.198.60" port: 38695 } has no permanent_uuid. Determining permanent_uuid...
I20260925 22:49:13.826773 968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:13.826454 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.198.60:38695 (local address 127.0.0.1:49254)
0925 22:49:13.826597 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0925 22:49:13.826699 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.198.60:38695: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260925 22:49:13.826997 912 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.10.198.60:38695: connect: Connection refused (error 111)
I20260925 22:49:13.828284 968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:13.828029 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.198.60:38695 (local address 127.0.0.1:49266)
0925 22:49:13.828148 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0925 22:49:13.828217 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.198.60:38695: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260925 22:49:13.828471 964 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.10.198.60:38695: Network error: Client connection negotiation failed: client connection to 127.10.198.60:38695: connect: Connection refused (error 111)
I20260925 22:49:13.840969 889 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.198.60" port: 38695 } attempt: 1
I20260925 22:49:13.841848 892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:13.841537 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.198.60:38695 (local address 127.0.0.1:49282)
0925 22:49:13.841696 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0925 22:49:13.841765 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.198.60:38695: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
I20260925 22:49:13.842823 892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:13.842566 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.198.60:38695 (local address 127.0.0.1:49296)
0925 22:49:13.842693 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0925 22:49:13.842755 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.198.60:38695: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260925 22:49:13.842978 889 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.10.198.60:38695: Network error: Client connection negotiation failed: client connection to 127.10.198.60:38695: connect: Connection refused (error 111)
I20260925 22:49:13.877784 964 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.198.60" port: 38695 } attempt: 1
I20260925 22:49:13.878750 968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:13.878428 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.198.60:38695 (local address 127.0.0.1:49308)
0925 22:49:13.878579 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0925 22:49:13.878653 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.198.60:38695: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
I20260925 22:49:13.879557 968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:13.879303 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.198.60:38695 (local address 127.0.0.1:49316)
0925 22:49:13.879449 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0925 22:49:13.879496 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.198.60:38695: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
W20260925 22:49:13.879694 964 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.10.198.60:38695: Network error: Client connection negotiation failed: client connection to 127.10.198.60:38695: connect: Connection refused (error 111)
I20260925 22:49:13.918183 889 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.198.60" port: 38695 } attempt: 2
I20260925 22:49:13.919186 892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:13.918835 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.198.60:38695 (local address 127.0.0.1:49322)
0925 22:49:13.919017 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0925 22:49:13.919095 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.198.60:38695: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81}
I20260925 22:49:13.920032 892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:13.919736 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.198.60:38695 (local address 127.0.0.1:49324)
0925 22:49:13.919881 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0925 22:49:13.919966 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.198.60:38695: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260925 22:49:13.920172 889 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.10.198.60:38695: Network error: Client connection negotiation failed: client connection to 127.10.198.60:38695: connect: Connection refused (error 111)
I20260925 22:49:13.948042 964 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.198.60" port: 38695 } attempt: 2
I20260925 22:49:13.948980 968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:13.948656 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.198.60:38695 (local address 127.0.0.1:49340)
0925 22:49:13.948838 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0925 22:49:13.948901 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.198.60:38695: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":90}
I20260925 22:49:13.949723 968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:13.949498 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.198.60:38695 (local address 127.0.0.1:49356)
0925 22:49:13.949612 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0925 22:49:13.949663 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.198.60:38695: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260925 22:49:13.949862 964 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.10.198.60:38695: Network error: Client connection negotiation failed: client connection to 127.10.198.60:38695: connect: Connection refused (error 111)
W20260925 22:49:13.952260 970 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260925 22:49:13.952663 970 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:49:13.952737 970 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:49:13.952859 970 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:49:13.957463 970 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 22:49:13.957604 970 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:49:13.957651 970 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:49:13.957715 970 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 22:49:13.957762 970 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 22:49:13.963704 970 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.198.20:33069
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790376259650642-11032-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790376259650642-11032-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.10.198.62:46393,127.10.198.61:46873,127.10.198.60:38695
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790376259650642-11032-0/krb5kdc/kudu_127.10.198.60.keytab
--principal=kudu/127.10.198.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.198.60:38695
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790376259650642-11032-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.10.198.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.970
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790376259650642-11032-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:09:16 UTC on ccycloud.quasar-afmhyw.root.comops.site
build id 4673379
I20260925 22:49:13.965063 970 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:49:13.966246 970 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:49:13.975726 977 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:49:13.975737 979 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:49:13.975755 976 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:49:13.976158 970 server_base.cc:1034] running on GCE node
I20260925 22:49:13.976837 970 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:49:13.978125 970 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:49:13.979374 970 hybrid_clock.cc:648] HybridClock initialized: now 1790376553979323 us; error 75 us; skew 500 ppm
Sep 25 22:49:13 dist-test-slave-5vdr krb5kdc[808](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376553, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.198.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:49:13.983124 970 init.cc:377] Logged in from keytab as kudu/127.10.198.60@KRBTEST.COM (short username kudu)
I20260925 22:49:13.984441 970 webserver.cc:460] Webserver started at http://127.10.198.60:33583/ using document root <none> and password file <none>
I20260925 22:49:13.985101 970 fs_manager.cc:359] Metadata directory not provided
I20260925 22:49:13.985227 970 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:49:13.985482 970 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:49:13.987929 970 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790376259650642-11032-0/minicluster-data/master-2/data/instance:
uuid: "86c5e1afa805401b81a5391886d1f898"
format_stamp: "Formatted at 2026-09-25 22:49:13 on dist-test-slave-5vdr"
server_key: "a23893b3d9266a5e5d5b95e316572df9"
server_key_iv: "2e5fe40e4f83a755ef01ce4b4030ecb1"
server_key_version: "encryptionkey@0"
I20260925 22:49:13.988638 970 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790376259650642-11032-0/minicluster-data/master-2/wal/instance:
uuid: "86c5e1afa805401b81a5391886d1f898"
format_stamp: "Formatted at 2026-09-25 22:49:13 on dist-test-slave-5vdr"
server_key: "a23893b3d9266a5e5d5b95e316572df9"
server_key_iv: "2e5fe40e4f83a755ef01ce4b4030ecb1"
server_key_version: "encryptionkey@0"
I20260925 22:49:13.993108 970 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260925 22:49:13.996081 986 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:49:13.997501 970 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 22:49:13.997709 970 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790376259650642-11032-0/minicluster-data/master-2/data,/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790376259650642-11032-0/minicluster-data/master-2/wal
uuid: "86c5e1afa805401b81a5391886d1f898"
format_stamp: "Formatted at 2026-09-25 22:49:13 on dist-test-slave-5vdr"
server_key: "a23893b3d9266a5e5d5b95e316572df9"
server_key_iv: "2e5fe40e4f83a755ef01ce4b4030ecb1"
server_key_version: "encryptionkey@0"
I20260925 22:49:13.997866 970 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790376259650642-11032-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790376259650642-11032-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790376259650642-11032-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:49:14.016187 970 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:49:14.019361 889 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.198.60" port: 38695 } attempt: 3
I20260925 22:49:14.019773 970 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:49:14.020022 970 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:49:14.020300 892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:14.019970 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.198.60:38695 (local address 127.0.0.1:49362)
0925 22:49:14.020139 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0925 22:49:14.020207 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.198.60:38695: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
I20260925 22:49:14.021059 892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:14.020831 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.198.60:38695 (local address 127.0.0.1:49368)
0925 22:49:14.020942 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0925 22:49:14.020992 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.198.60:38695: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260925 22:49:14.021219 889 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.10.198.60:38695: Network error: Client connection negotiation failed: client connection to 127.10.198.60:38695: connect: Connection refused (error 111)
I20260925 22:49:14.027932 970 rpc_server.cc:307] RPC server started. Bound to: 127.10.198.60:38695
I20260925 22:49:14.028254 1038 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.198.60:38695 every 8 connection(s)
I20260925 22:49:14.029129 970 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790376259650642-11032-0/minicluster-data/master-2/data/info.pb
I20260925 22:49:14.032474 1039 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:49:14.032735 11032 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrWXm8X/build/debug/bin/kudu as pid 970
I20260925 22:49:14.032892 11032 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790376259650642-11032-0/minicluster-data/master-2/wal/instance
I20260925 22:49:14.033205 11032 external_mini_cluster.cc:1428] Setting key 8812b999f30c40747771bfc93c7d07d3
I20260925 22:49:14.038235 1039 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.198.62" port: 46393 } has no permanent_uuid. Determining permanent_uuid...
I20260925 22:49:14.041036 964 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.198.60" port: 38695 } attempt: 3
Sep 25 22:49:14 dist-test-slave-5vdr krb5kdc[808](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376553, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.198.60@KRBTEST.COM
Sep 25 22:49:14 dist-test-slave-5vdr krb5kdc[808](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376553, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.198.61@KRBTEST.COM for kudu/127.10.198.60@KRBTEST.COM
Sep 25 22:49:14 dist-test-slave-5vdr krb5kdc[808](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376553, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.198.60@KRBTEST.COM for kudu/127.10.198.62@KRBTEST.COM
I20260925 22:49:14.053149 893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:14.040642 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41856 (local address 127.10.198.62:46393)
0925 22:49:14.040797 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0925 22:49:14.040801 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:49:14.041370 (+ 569us) server_negotiation.cc:407] Connection header received
0925 22:49:14.041627 (+ 257us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:49:14.041630 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:49:14.041691 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:49:14.041755 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:49:14.042996 (+ 1241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:49:14.043625 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:49:14.044807 (+ 1182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:49:14.045194 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:49:14.045342 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:49:14.049105 (+ 3763us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:49:14.049108 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:49:14.049111 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:49:14.049140 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:49:14.050835 (+ 1695us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:49:14.051591 (+ 756us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:49:14.051595 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:49:14.051597 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:49:14.051698 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:49:14.052057 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:49:14.052060 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:49:14.052062 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:49:14.052299 (+ 237us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:49:14.052375 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0925 22:49:14.052837 (+ 462us) server_negotiation.cc:299] Negotiation successful
0925 22:49:14.052982 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260925 22:49:14.053400 968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:14.041529 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.198.60:38695 (local address 127.0.0.1:49388)
0925 22:49:14.041790 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0925 22:49:14.041816 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0925 22:49:14.041913 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:49:14.042533 (+ 620us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:49:14.042537 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:49:14.042799 (+ 262us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:49:14.042937 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:49:14.042949 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:49:14.043975 (+ 1026us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:49:14.043980 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:49:14.044905 (+ 925us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:49:14.044921 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:49:14.045536 (+ 615us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:49:14.045540 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:49:14.045660 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:49:14.046289 (+ 629us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:49:14.046307 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:49:14.048094 (+ 1787us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:49:14.051678 (+ 3584us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:49:14.051682 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:49:14.051685 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:49:14.051999 (+ 314us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:49:14.052324 (+ 325us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:49:14.052328 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:49:14.052330 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:49:14.052426 (+ 96us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:49:14.052812 (+ 386us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:49:14.052815 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:49:14.052924 (+ 109us) client_negotiation.cc:769] Sending connection context
0925 22:49:14.053044 (+ 120us) client_negotiation.cc:240] Negotiation successful
0925 22:49:14.053203 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":151}
I20260925 22:49:14.053479 1042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:14.034826 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49382 (local address 127.10.198.60:38695)
0925 22:49:14.035356 (+ 530us) server_negotiation.cc:206] Beginning negotiation
0925 22:49:14.035377 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0925 22:49:14.035429 (+ 52us) server_negotiation.cc:407] Connection header received
0925 22:49:14.036332 (+ 903us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:49:14.036363 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:49:14.036884 (+ 521us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:49:14.037338 (+ 454us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:49:14.038189 (+ 851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:49:14.039682 (+ 1493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:49:14.041127 (+ 1445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:49:14.041962 (+ 835us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:49:14.042138 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:49:14.046359 (+ 4221us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:49:14.046369 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:49:14.046388 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:49:14.046420 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:49:14.048856 (+ 2436us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:49:14.049422 (+ 566us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:49:14.049429 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:49:14.049437 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:49:14.049631 (+ 194us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:49:14.050312 (+ 681us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:49:14.050317 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:49:14.050318 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:49:14.051024 (+ 706us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:49:14.051260 (+ 236us) server_negotiation.cc:1035] Waiting for connection context
0925 22:49:14.051653 (+ 393us) server_negotiation.cc:299] Negotiation successful
0925 22:49:14.052078 (+ 425us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":360,"thread_start_us":171,"threads_started":1}
I20260925 22:49:14.053992 1044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:14.041792 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49388 (local address 127.10.198.60:38695)
0925 22:49:14.042134 (+ 342us) server_negotiation.cc:206] Beginning negotiation
0925 22:49:14.042142 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 22:49:14.042154 (+ 12us) server_negotiation.cc:407] Connection header received
0925 22:49:14.042186 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:49:14.042189 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:49:14.042233 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:49:14.042512 (+ 279us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:49:14.043086 (+ 574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:49:14.043813 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:49:14.045048 (+ 1235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:49:14.045365 (+ 317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:49:14.045508 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:49:14.048431 (+ 2923us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:49:14.048435 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:49:14.048437 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:49:14.048468 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:49:14.051524 (+ 3056us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:49:14.052129 (+ 605us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:49:14.052131 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:49:14.052132 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:49:14.052222 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:49:14.052527 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:49:14.052532 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:49:14.052534 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:49:14.052708 (+ 174us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:49:14.052820 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0925 22:49:14.053097 (+ 277us) server_negotiation.cc:299] Negotiation successful
0925 22:49:14.053849 (+ 752us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":275,"thread_start_us":135,"threads_started":1}
I20260925 22:49:14.054155 1043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:14.040606 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.198.62:46393 (local address 127.0.0.1:41856)
0925 22:49:14.041214 (+ 608us) negotiation.cc:107] Waiting for socket to connect
0925 22:49:14.041286 (+ 72us) client_negotiation.cc:174] Beginning negotiation
0925 22:49:14.041497 (+ 211us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:49:14.041990 (+ 493us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:49:14.041999 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:49:14.042647 (+ 648us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:49:14.042843 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:49:14.042851 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:49:14.043803 (+ 952us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:49:14.043807 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:49:14.044630 (+ 823us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:49:14.044642 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:49:14.045413 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:49:14.045421 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:49:14.045546 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:49:14.046196 (+ 650us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:49:14.046225 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:49:14.048856 (+ 2631us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:49:14.050985 (+ 2129us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:49:14.050994 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:49:14.051000 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:49:14.051462 (+ 462us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:49:14.051817 (+ 355us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:49:14.051820 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:49:14.051822 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:49:14.051937 (+ 115us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:49:14.052468 (+ 531us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:49:14.052474 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:49:14.052601 (+ 127us) client_negotiation.cc:769] Sending connection context
0925 22:49:14.053383 (+ 782us) client_negotiation.cc:240] Negotiation successful
0925 22:49:14.053955 (+ 572us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":351,"mutex_wait_us":325,"thread_start_us":201,"threads_started":1}
I20260925 22:49:14.055909 1039 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.198.61" port: 46873 } has no permanent_uuid. Determining permanent_uuid...
I20260925 22:49:14.058178 964 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f3fc2bc68b2f4752a289c342d2a82bad: Bootstrap starting.
I20260925 22:49:14.061044 964 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f3fc2bc68b2f4752a289c342d2a82bad: Neither blocks nor log segments found. Creating new log.
I20260925 22:49:14.062043 964 log.cc:824] T 00000000000000000000000000000000 P f3fc2bc68b2f4752a289c342d2a82bad: Log is configured to *not* fsync() on all Append() calls
I20260925 22:49:14.065086 964 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f3fc2bc68b2f4752a289c342d2a82bad: No bootstrap required, opened a new log
Sep 25 22:49:14 dist-test-slave-5vdr krb5kdc[808](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376553, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.198.60@KRBTEST.COM for kudu/127.10.198.61@KRBTEST.COM
Sep 25 22:49:14 dist-test-slave-5vdr krb5kdc[808](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376553, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.198.62@KRBTEST.COM
I20260925 22:49:14.070219 964 raft_consensus.cc:348] T 00000000000000000000000000000000 P f3fc2bc68b2f4752a289c342d2a82bad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f6b93a17812478199d694a247f1af35" member_type: VOTER last_known_addr { host: "127.10.198.62" port: 46393 } } peers { permanent_uuid: "f3fc2bc68b2f4752a289c342d2a82bad" member_type: VOTER last_known_addr { host: "127.10.198.61" port: 46873 } } peers { permanent_uuid: "86c5e1afa805401b81a5391886d1f898" member_type: VOTER last_known_addr { host: "127.10.198.60" port: 38695 } }
I20260925 22:49:14.070559 964 raft_consensus.cc:729] T 00000000000000000000000000000000 P f3fc2bc68b2f4752a289c342d2a82bad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f3fc2bc68b2f4752a289c342d2a82bad, State: Initialized, Role: FOLLOWER
I20260925 22:49:14.070955 964 consensus_queue.cc:260] T 00000000000000000000000000000000 P f3fc2bc68b2f4752a289c342d2a82bad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f6b93a17812478199d694a247f1af35" member_type: VOTER last_known_addr { host: "127.10.198.62" port: 46393 } } peers { permanent_uuid: "f3fc2bc68b2f4752a289c342d2a82bad" member_type: VOTER last_known_addr { host: "127.10.198.61" port: 46873 } } peers { permanent_uuid: "86c5e1afa805401b81a5391886d1f898" member_type: VOTER last_known_addr { host: "127.10.198.60" port: 38695 } }
I20260925 22:49:14.071563 1046 sys_catalog.cc:455] T 00000000000000000000000000000000 P f3fc2bc68b2f4752a289c342d2a82bad [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f6b93a17812478199d694a247f1af35" member_type: VOTER last_known_addr { host: "127.10.198.62" port: 46393 } } peers { permanent_uuid: "f3fc2bc68b2f4752a289c342d2a82bad" member_type: VOTER last_known_addr { host: "127.10.198.61" port: 46873 } } peers { permanent_uuid: "86c5e1afa805401b81a5391886d1f898" member_type: VOTER last_known_addr { host: "127.10.198.60" port: 38695 } } }
I20260925 22:49:14.071882 966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:14.060193 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54422 (local address 127.10.198.61:46873)
0925 22:49:14.060460 (+ 267us) server_negotiation.cc:206] Beginning negotiation
0925 22:49:14.060465 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:49:14.060484 (+ 19us) server_negotiation.cc:407] Connection header received
0925 22:49:14.060535 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:49:14.060539 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:49:14.060590 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:49:14.060692 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:49:14.061592 (+ 900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:49:14.062475 (+ 883us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:49:14.063780 (+ 1305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:49:14.064140 (+ 360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:49:14.064256 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:49:14.066908 (+ 2652us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:49:14.066913 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:49:14.066916 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:49:14.066950 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:49:14.069493 (+ 2543us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:49:14.070149 (+ 656us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:49:14.070153 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:49:14.070155 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:49:14.070284 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:49:14.070691 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:49:14.070707 (+ 16us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:49:14.070712 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:49:14.071013 (+ 301us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:49:14.071082 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0925 22:49:14.071506 (+ 424us) server_negotiation.cc:299] Negotiation successful
0925 22:49:14.071660 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":169}
I20260925 22:49:14.071915 893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:14.056647 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41868 (local address 127.10.198.62:46393)
0925 22:49:14.056801 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0925 22:49:14.056805 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:49:14.060499 (+ 3694us) server_negotiation.cc:407] Connection header received
0925 22:49:14.060695 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:49:14.060698 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:49:14.060760 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:49:14.060859 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:49:14.061744 (+ 885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:49:14.062458 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:49:14.063566 (+ 1108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:49:14.063955 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:49:14.064070 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:49:14.067742 (+ 3672us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:49:14.067748 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:49:14.067751 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:49:14.067791 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:49:14.069558 (+ 1767us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:49:14.070188 (+ 630us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:49:14.070191 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:49:14.070193 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:49:14.070312 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:49:14.070730 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:49:14.070734 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:49:14.070735 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:49:14.070971 (+ 236us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:49:14.071216 (+ 245us) server_negotiation.cc:1035] Waiting for connection context
0925 22:49:14.071618 (+ 402us) server_negotiation.cc:299] Negotiation successful
0925 22:49:14.071745 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260925 22:49:14.072243 1043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:14.056470 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.198.61:46873 (local address 127.0.0.1:54422)
0925 22:49:14.056623 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0925 22:49:14.056640 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 22:49:14.056734 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:49:14.061021 (+ 4287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:49:14.061025 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:49:14.061237 (+ 212us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:49:14.061427 (+ 190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:49:14.061437 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:49:14.062731 (+ 1294us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:49:14.062735 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:49:14.063616 (+ 881us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:49:14.063635 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:49:14.064273 (+ 638us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:49:14.064276 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:49:14.064385 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:49:14.065010 (+ 625us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:49:14.065032 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:49:14.066666 (+ 1634us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:49:14.069688 (+ 3022us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:49:14.069693 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:49:14.069696 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:49:14.070014 (+ 318us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:49:14.070434 (+ 420us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:49:14.070440 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:49:14.070441 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:49:14.070556 (+ 115us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:49:14.071169 (+ 613us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:49:14.071173 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:49:14.071301 (+ 128us) client_negotiation.cc:769] Sending connection context
0925 22:49:14.071909 (+ 608us) client_negotiation.cc:240] Negotiation successful
0925 22:49:14.072087 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30}
I20260925 22:49:14.071878 1046 sys_catalog.cc:458] T 00000000000000000000000000000000 P f3fc2bc68b2f4752a289c342d2a82bad [sys.catalog]: This master's current role is: FOLLOWER
I20260925 22:49:14.072834 964 sys_catalog.cc:564] T 00000000000000000000000000000000 P f3fc2bc68b2f4752a289c342d2a82bad [sys.catalog]: configured and running, proceeding with master startup.
I20260925 22:49:14.073359 1039 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.198.60" port: 38695 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 22:49:14 dist-test-slave-5vdr krb5kdc[808](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376553, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.198.60@KRBTEST.COM for kudu/127.10.198.60@KRBTEST.COM
W20260925 22:49:14.079970 1057 catalog_manager.cc:1558] T 00000000000000000000000000000000 P f3fc2bc68b2f4752a289c342d2a82bad: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 22:49:14.080060 1057 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
Sep 25 22:49:14 dist-test-slave-5vdr krb5kdc[808](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376553, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.198.61@KRBTEST.COM
I20260925 22:49:14.077529 896 master_runner.cc:186] Error getting master registration for 127.10.198.62:46393: OK, instance_id { permanent_uuid: "2f6b93a17812478199d694a247f1af35" instance_seqno: 1790376553534903 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260925 22:49:14.082602 896 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260925 22:49:14.084004 1043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:14.073933 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.198.60:38695 (local address 127.0.0.1:49392)
0925 22:49:14.074041 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0925 22:49:14.074059 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0925 22:49:14.074152 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:49:14.074460 (+ 308us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:49:14.074463 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:49:14.074668 (+ 205us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:49:14.074755 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:49:14.074761 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:49:14.075734 (+ 973us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:49:14.075738 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:49:14.076520 (+ 782us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:49:14.076531 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:49:14.077208 (+ 677us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:49:14.077211 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:49:14.077288 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:49:14.077850 (+ 562us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:49:14.077865 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:49:14.079475 (+ 1610us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:49:14.081918 (+ 2443us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:49:14.081921 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:49:14.081923 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:49:14.082167 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:49:14.082764 (+ 597us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:49:14.082767 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:49:14.082769 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:49:14.082855 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:49:14.083379 (+ 524us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:49:14.083381 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:49:14.083502 (+ 121us) client_negotiation.cc:769] Sending connection context
0925 22:49:14.083619 (+ 117us) client_negotiation.cc:240] Negotiation successful
0925 22:49:14.083788 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32}
I20260925 22:49:14.084208 1044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:14.074168 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49392 (local address 127.10.198.60:38695)
0925 22:49:14.074270 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0925 22:49:14.074274 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:49:14.074284 (+ 10us) server_negotiation.cc:407] Connection header received
0925 22:49:14.074322 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:49:14.074324 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:49:14.074360 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:49:14.074453 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:49:14.074875 (+ 422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:49:14.075593 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:49:14.076649 (+ 1056us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:49:14.076979 (+ 330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:49:14.077077 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:49:14.079632 (+ 2555us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:49:14.079635 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:49:14.079638 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:49:14.079660 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:49:14.081783 (+ 2123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:49:14.082279 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:49:14.082281 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:49:14.082283 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:49:14.082642 (+ 359us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:49:14.082944 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:49:14.082947 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:49:14.082948 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:49:14.083243 (+ 295us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:49:14.083342 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0925 22:49:14.083750 (+ 408us) server_negotiation.cc:299] Negotiation successful
0925 22:49:14.083960 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260925 22:49:14.087384 966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:14.074557 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54428 (local address 127.10.198.61:46873)
0925 22:49:14.074679 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0925 22:49:14.074684 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:49:14.074699 (+ 15us) server_negotiation.cc:407] Connection header received
0925 22:49:14.074744 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:49:14.074750 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:49:14.074809 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:49:14.075918 (+ 1109us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:49:14.076036 (+ 118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:49:14.076694 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:49:14.077820 (+ 1126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:49:14.078191 (+ 371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:49:14.078726 (+ 535us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:49:14.081986 (+ 3260us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:49:14.081991 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:49:14.081995 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:49:14.082030 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:49:14.084533 (+ 2503us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:49:14.085218 (+ 685us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:49:14.085223 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:49:14.085224 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:49:14.085344 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:49:14.085745 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:49:14.085749 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:49:14.085751 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:49:14.085998 (+ 247us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:49:14.086936 (+ 938us) server_negotiation.cc:1035] Waiting for connection context
0925 22:49:14.087037 (+ 101us) server_negotiation.cc:299] Negotiation successful
0925 22:49:14.087198 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260925 22:49:14.088037 1039 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 86c5e1afa805401b81a5391886d1f898: Bootstrap starting.
I20260925 22:49:14.091341 1039 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 86c5e1afa805401b81a5391886d1f898: Neither blocks nor log segments found. Creating new log.
I20260925 22:49:14.092157 11032 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 22:49:14.092466 1039 log.cc:824] T 00000000000000000000000000000000 P 86c5e1afa805401b81a5391886d1f898: Log is configured to *not* fsync() on all Append() calls
I20260925 22:49:14.095136 1039 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 86c5e1afa805401b81a5391886d1f898: No bootstrap required, opened a new log
I20260925 22:49:14.093794 863 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "2f6b93a17812478199d694a247f1af35"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41868
I20260925 22:49:14.100260 1039 raft_consensus.cc:348] T 00000000000000000000000000000000 P 86c5e1afa805401b81a5391886d1f898 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f6b93a17812478199d694a247f1af35" member_type: VOTER last_known_addr { host: "127.10.198.62" port: 46393 } } peers { permanent_uuid: "f3fc2bc68b2f4752a289c342d2a82bad" member_type: VOTER last_known_addr { host: "127.10.198.61" port: 46873 } } peers { permanent_uuid: "86c5e1afa805401b81a5391886d1f898" member_type: VOTER last_known_addr { host: "127.10.198.60" port: 38695 } }
I20260925 22:49:14.100638 1039 raft_consensus.cc:729] T 00000000000000000000000000000000 P 86c5e1afa805401b81a5391886d1f898 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 86c5e1afa805401b81a5391886d1f898, State: Initialized, Role: FOLLOWER
I20260925 22:49:14.101052 1039 consensus_queue.cc:260] T 00000000000000000000000000000000 P 86c5e1afa805401b81a5391886d1f898 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f6b93a17812478199d694a247f1af35" member_type: VOTER last_known_addr { host: "127.10.198.62" port: 46393 } } peers { permanent_uuid: "f3fc2bc68b2f4752a289c342d2a82bad" member_type: VOTER last_known_addr { host: "127.10.198.61" port: 46873 } } peers { permanent_uuid: "86c5e1afa805401b81a5391886d1f898" member_type: VOTER last_known_addr { host: "127.10.198.60" port: 38695 } }
I20260925 22:49:14.101711 1060 sys_catalog.cc:455] T 00000000000000000000000000000000 P 86c5e1afa805401b81a5391886d1f898 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f6b93a17812478199d694a247f1af35" member_type: VOTER last_known_addr { host: "127.10.198.62" port: 46393 } } peers { permanent_uuid: "f3fc2bc68b2f4752a289c342d2a82bad" member_type: VOTER last_known_addr { host: "127.10.198.61" port: 46873 } } peers { permanent_uuid: "86c5e1afa805401b81a5391886d1f898" member_type: VOTER last_known_addr { host: "127.10.198.60" port: 38695 } } }
I20260925 22:49:14.101936 1060 sys_catalog.cc:458] T 00000000000000000000000000000000 P 86c5e1afa805401b81a5391886d1f898 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 22:49:14.102844 1039 sys_catalog.cc:564] T 00000000000000000000000000000000 P 86c5e1afa805401b81a5391886d1f898 [sys.catalog]: configured and running, proceeding with master startup.
W20260925 22:49:14.109675 1071 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 86c5e1afa805401b81a5391886d1f898: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 22:49:14.109825 1071 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 22:49:14.107336 970 master_runner.cc:186] Error getting master registration for 127.10.198.62:46393: OK, instance_id { permanent_uuid: "2f6b93a17812478199d694a247f1af35" instance_seqno: 1790376553534903 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260925 22:49:14.111524 970 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260925 22:49:14.185375 889 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.198.60" port: 38695 } attempt: 4
Sep 25 22:49:14 dist-test-slave-5vdr krb5kdc[808](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376553, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.198.62@KRBTEST.COM for kudu/127.10.198.60@KRBTEST.COM
I20260925 22:49:14.197645 892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:14.186077 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.198.60:38695 (local address 127.0.0.1:49404)
0925 22:49:14.186339 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0925 22:49:14.186358 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0925 22:49:14.186485 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:49:14.186822 (+ 337us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:49:14.186826 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:49:14.187085 (+ 259us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:49:14.187245 (+ 160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:49:14.187255 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:49:14.188838 (+ 1583us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:49:14.188843 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:49:14.189855 (+ 1012us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:49:14.189871 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:49:14.190530 (+ 659us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:49:14.190533 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:49:14.190635 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:49:14.191240 (+ 605us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:49:14.191258 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:49:14.193174 (+ 1916us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:49:14.195571 (+ 2397us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:49:14.195576 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:49:14.195578 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:49:14.195948 (+ 370us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:49:14.196340 (+ 392us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:49:14.196344 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:49:14.196346 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:49:14.196449 (+ 103us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:49:14.197025 (+ 576us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:49:14.197029 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:49:14.197156 (+ 127us) client_negotiation.cc:769] Sending connection context
0925 22:49:14.197276 (+ 120us) client_negotiation.cc:240] Negotiation successful
0925 22:49:14.197454 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":170}
I20260925 22:49:14.198312 1044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:14.186164 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49404 (local address 127.10.198.60:38695)
0925 22:49:14.186314 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0925 22:49:14.186319 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:49:14.186445 (+ 126us) server_negotiation.cc:407] Connection header received
0925 22:49:14.186593 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:49:14.186597 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:49:14.186657 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:49:14.186768 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:49:14.187528 (+ 760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:49:14.188641 (+ 1113us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:49:14.190046 (+ 1405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:49:14.190382 (+ 336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:49:14.190512 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:49:14.193410 (+ 2898us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:49:14.193415 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:49:14.193419 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:49:14.193460 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:49:14.195380 (+ 1920us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:49:14.196102 (+ 722us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:49:14.196106 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:49:14.196109 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:49:14.196217 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:49:14.196565 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:49:14.196569 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:49:14.196571 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:49:14.196882 (+ 311us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:49:14.197011 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0925 22:49:14.198011 (+ 1000us) server_negotiation.cc:299] Negotiation successful
0925 22:49:14.198153 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260925 22:49:14.202455 889 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2f6b93a17812478199d694a247f1af35: Bootstrap starting.
I20260925 22:49:14.206200 889 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2f6b93a17812478199d694a247f1af35: Neither blocks nor log segments found. Creating new log.
I20260925 22:49:14.207345 889 log.cc:824] T 00000000000000000000000000000000 P 2f6b93a17812478199d694a247f1af35: Log is configured to *not* fsync() on all Append() calls
I20260925 22:49:14.211131 889 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2f6b93a17812478199d694a247f1af35: No bootstrap required, opened a new log
I20260925 22:49:14.216454 889 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2f6b93a17812478199d694a247f1af35 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f6b93a17812478199d694a247f1af35" member_type: VOTER last_known_addr { host: "127.10.198.62" port: 46393 } } peers { permanent_uuid: "f3fc2bc68b2f4752a289c342d2a82bad" member_type: VOTER last_known_addr { host: "127.10.198.61" port: 46873 } } peers { permanent_uuid: "86c5e1afa805401b81a5391886d1f898" member_type: VOTER last_known_addr { host: "127.10.198.60" port: 38695 } }
I20260925 22:49:14.217020 889 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2f6b93a17812478199d694a247f1af35 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f6b93a17812478199d694a247f1af35, State: Initialized, Role: FOLLOWER
I20260925 22:49:14.217680 889 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2f6b93a17812478199d694a247f1af35 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f6b93a17812478199d694a247f1af35" member_type: VOTER last_known_addr { host: "127.10.198.62" port: 46393 } } peers { permanent_uuid: "f3fc2bc68b2f4752a289c342d2a82bad" member_type: VOTER last_known_addr { host: "127.10.198.61" port: 46873 } } peers { permanent_uuid: "86c5e1afa805401b81a5391886d1f898" member_type: VOTER last_known_addr { host: "127.10.198.60" port: 38695 } }
I20260925 22:49:14.218534 1074 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2f6b93a17812478199d694a247f1af35 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f6b93a17812478199d694a247f1af35" member_type: VOTER last_known_addr { host: "127.10.198.62" port: 46393 } } peers { permanent_uuid: "f3fc2bc68b2f4752a289c342d2a82bad" member_type: VOTER last_known_addr { host: "127.10.198.61" port: 46873 } } peers { permanent_uuid: "86c5e1afa805401b81a5391886d1f898" member_type: VOTER last_known_addr { host: "127.10.198.60" port: 38695 } } }
I20260925 22:49:14.218701 1074 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2f6b93a17812478199d694a247f1af35 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 22:49:14.220427 863 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2f6b93a17812478199d694a247f1af35 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260925 22:49:14.220563 863 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2f6b93a17812478199d694a247f1af35 [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:49:14.221987 863 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2f6b93a17812478199d694a247f1af35 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f6b93a17812478199d694a247f1af35" member_type: VOTER last_known_addr { host: "127.10.198.62" port: 46393 } } peers { permanent_uuid: "f3fc2bc68b2f4752a289c342d2a82bad" member_type: VOTER last_known_addr { host: "127.10.198.61" port: 46873 } } peers { permanent_uuid: "86c5e1afa805401b81a5391886d1f898" member_type: VOTER last_known_addr { host: "127.10.198.60" port: 38695 } }
I20260925 22:49:14.223526 863 leader_election.cc:288] T 00000000000000000000000000000000 P 2f6b93a17812478199d694a247f1af35 [CANDIDATE]: Term 1 election: Requested vote from peers f3fc2bc68b2f4752a289c342d2a82bad (127.10.198.61:46873), 86c5e1afa805401b81a5391886d1f898 (127.10.198.60:38695)
I20260925 22:49:14.223871 889 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2f6b93a17812478199d694a247f1af35 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 22:49:14.224869 938 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "2f6b93a17812478199d694a247f1af35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "f3fc2bc68b2f4752a289c342d2a82bad"
I20260925 22:49:14.225212 938 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f3fc2bc68b2f4752a289c342d2a82bad [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:49:14.226846 938 raft_consensus.cc:2454] T 00000000000000000000000000000000 P f3fc2bc68b2f4752a289c342d2a82bad [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2f6b93a17812478199d694a247f1af35 in term 1.
I20260925 22:49:14.227519 1013 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "2f6b93a17812478199d694a247f1af35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "86c5e1afa805401b81a5391886d1f898"
I20260925 22:49:14.228227 837 leader_election.cc:302] T 00000000000000000000000000000000 P 2f6b93a17812478199d694a247f1af35 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2f6b93a17812478199d694a247f1af35, f3fc2bc68b2f4752a289c342d2a82bad; no voters:
I20260925 22:49:14.228559 1013 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 86c5e1afa805401b81a5391886d1f898 [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:49:14.230298 1013 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 86c5e1afa805401b81a5391886d1f898 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2f6b93a17812478199d694a247f1af35 in term 1.
W20260925 22:49:14.231662 1085 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 2f6b93a17812478199d694a247f1af35: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 22:49:14.231837 1085 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 22:49:14.234712 1074 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2f6b93a17812478199d694a247f1af35 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 22:49:14.235037 1074 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2f6b93a17812478199d694a247f1af35 [term 1 LEADER]: Becoming Leader. State: Replica: 2f6b93a17812478199d694a247f1af35, State: Running, Role: LEADER
I20260925 22:49:14.235467 1074 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2f6b93a17812478199d694a247f1af35 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f6b93a17812478199d694a247f1af35" member_type: VOTER last_known_addr { host: "127.10.198.62" port: 46393 } } peers { permanent_uuid: "f3fc2bc68b2f4752a289c342d2a82bad" member_type: VOTER last_known_addr { host: "127.10.198.61" port: 46873 } } peers { permanent_uuid: "86c5e1afa805401b81a5391886d1f898" member_type: VOTER last_known_addr { host: "127.10.198.60" port: 38695 } }
I20260925 22:49:14.237356 821 master_runner.cc:263] No leader master found from master 2f6b93a17812478199d694a247f1af35
I20260925 22:49:14.237460 821 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260925 22:49:14.239251 1074 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2f6b93a17812478199d694a247f1af35 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2f6b93a17812478199d694a247f1af35. Latest consensus state: current_term: 1 leader_uuid: "2f6b93a17812478199d694a247f1af35" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f6b93a17812478199d694a247f1af35" member_type: VOTER last_known_addr { host: "127.10.198.62" port: 46393 } } peers { permanent_uuid: "f3fc2bc68b2f4752a289c342d2a82bad" member_type: VOTER last_known_addr { host: "127.10.198.61" port: 46873 } } peers { permanent_uuid: "86c5e1afa805401b81a5391886d1f898" member_type: VOTER last_known_addr { host: "127.10.198.60" port: 38695 } } }
I20260925 22:49:14.239452 1074 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2f6b93a17812478199d694a247f1af35 [sys.catalog]: This master's current role is: LEADER
I20260925 22:49:14.244175 1093 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 22:49:14.245328 1093 catalog_manager.cc:1484] Initializing Kudu cluster ID...
W20260925 22:49:14.271886 992 tablet.cc:2369] T 00000000000000000000000000000000 P 86c5e1afa805401b81a5391886d1f898: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 22:49:14.277892 917 tablet.cc:2369] T 00000000000000000000000000000000 P f3fc2bc68b2f4752a289c342d2a82bad: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 22:49:14.280078 842 tablet.cc:2369] T 00000000000000000000000000000000 P 2f6b93a17812478199d694a247f1af35: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 22:49:14.283332 1044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:14.242110 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49418 (local address 127.10.198.60:38695)
0925 22:49:14.242243 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0925 22:49:14.242248 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:49:14.250281 (+ 8033us) server_negotiation.cc:407] Connection header received
0925 22:49:14.253544 (+ 3263us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:49:14.253552 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:49:14.253620 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:49:14.253709 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:49:14.255261 (+ 1552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:49:14.256145 (+ 884us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:49:14.264461 (+ 8316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:49:14.264825 (+ 364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:49:14.264931 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:49:14.270525 (+ 5594us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:49:14.270531 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:49:14.270534 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:49:14.270578 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:49:14.272655 (+ 2077us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:49:14.279018 (+ 6363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:49:14.279024 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:49:14.279026 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:49:14.279144 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:49:14.279938 (+ 794us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:49:14.279943 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:49:14.279945 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:49:14.280215 (+ 270us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:49:14.280383 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0925 22:49:14.282950 (+ 2567us) server_negotiation.cc:299] Negotiation successful
0925 22:49:14.283128 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260925 22:49:14.285908 966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:14.240322 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54432 (local address 127.10.198.61:46873)
0925 22:49:14.240473 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0925 22:49:14.240479 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 22:49:14.250020 (+ 9541us) server_negotiation.cc:407] Connection header received
0925 22:49:14.250560 (+ 540us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:49:14.250564 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:49:14.250629 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:49:14.250719 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:49:14.256426 (+ 5707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:49:14.257172 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:49:14.263769 (+ 6597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:49:14.264148 (+ 379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:49:14.264259 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:49:14.272977 (+ 8718us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:49:14.272983 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:49:14.272986 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:49:14.273027 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:49:14.275165 (+ 2138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:49:14.279512 (+ 4347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:49:14.279518 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:49:14.279520 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:49:14.279626 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:49:14.281343 (+ 1717us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:49:14.281347 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:49:14.281349 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:49:14.281605 (+ 256us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:49:14.281747 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0925 22:49:14.285524 (+ 3777us) server_negotiation.cc:299] Negotiation successful
0925 22:49:14.285708 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260925 22:49:14.288374 893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:14.240870 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41880 (local address 127.10.198.62:46393)
0925 22:49:14.241014 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0925 22:49:14.241020 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 22:49:14.249744 (+ 8724us) server_negotiation.cc:407] Connection header received
0925 22:49:14.252238 (+ 2494us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:49:14.252243 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:49:14.252311 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:49:14.252379 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:49:14.257382 (+ 5003us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:49:14.258163 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:49:14.262308 (+ 4145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:49:14.262661 (+ 353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:49:14.262786 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:49:14.276440 (+ 13654us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:49:14.276445 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:49:14.276448 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:49:14.276491 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:49:14.278347 (+ 1856us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:49:14.280783 (+ 2436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:49:14.280788 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:49:14.280790 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:49:14.280907 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:49:14.282105 (+ 1198us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:49:14.282110 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:49:14.282111 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:49:14.282372 (+ 261us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:49:14.282516 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0925 22:49:14.287944 (+ 5428us) server_negotiation.cc:299] Negotiation successful
0925 22:49:14.288133 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260925 22:49:14.300354 1013 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 86c5e1afa805401b81a5391886d1f898 [term 1 FOLLOWER]: Refusing update from remote peer 2f6b93a17812478199d694a247f1af35: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 22:49:14.303268 1091 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 2f6b93a17812478199d694a247f1af35 [LEADER]: Connected to new peer: Peer: permanent_uuid: "86c5e1afa805401b81a5391886d1f898" member_type: VOTER last_known_addr { host: "127.10.198.60" port: 38695 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 22:49:14.309096 1060 sys_catalog.cc:455] T 00000000000000000000000000000000 P 86c5e1afa805401b81a5391886d1f898 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2f6b93a17812478199d694a247f1af35. Latest consensus state: current_term: 1 leader_uuid: "2f6b93a17812478199d694a247f1af35" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f6b93a17812478199d694a247f1af35" member_type: VOTER last_known_addr { host: "127.10.198.62" port: 46393 } } peers { permanent_uuid: "f3fc2bc68b2f4752a289c342d2a82bad" member_type: VOTER last_known_addr { host: "127.10.198.61" port: 46873 } } peers { permanent_uuid: "86c5e1afa805401b81a5391886d1f898" member_type: VOTER last_known_addr { host: "127.10.198.60" port: 38695 } } }
I20260925 22:49:14.309322 1060 sys_catalog.cc:458] T 00000000000000000000000000000000 P 86c5e1afa805401b81a5391886d1f898 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 22:49:14.310926 938 raft_consensus.cc:1261] T 00000000000000000000000000000000 P f3fc2bc68b2f4752a289c342d2a82bad [term 1 FOLLOWER]: Refusing update from remote peer 2f6b93a17812478199d694a247f1af35: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 22:49:14.312157 1091 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 2f6b93a17812478199d694a247f1af35 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f3fc2bc68b2f4752a289c342d2a82bad" member_type: VOTER last_known_addr { host: "127.10.198.61" port: 46873 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 22:49:14.320264 1046 sys_catalog.cc:455] T 00000000000000000000000000000000 P f3fc2bc68b2f4752a289c342d2a82bad [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2f6b93a17812478199d694a247f1af35. Latest consensus state: current_term: 1 leader_uuid: "2f6b93a17812478199d694a247f1af35" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f6b93a17812478199d694a247f1af35" member_type: VOTER last_known_addr { host: "127.10.198.62" port: 46393 } } peers { permanent_uuid: "f3fc2bc68b2f4752a289c342d2a82bad" member_type: VOTER last_known_addr { host: "127.10.198.61" port: 46873 } } peers { permanent_uuid: "86c5e1afa805401b81a5391886d1f898" member_type: VOTER last_known_addr { host: "127.10.198.60" port: 38695 } } }
I20260925 22:49:14.320495 1046 sys_catalog.cc:458] T 00000000000000000000000000000000 P f3fc2bc68b2f4752a289c342d2a82bad [sys.catalog]: This master's current role is: FOLLOWER
I20260925 22:49:14.321352 1091 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2f6b93a17812478199d694a247f1af35 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "2f6b93a17812478199d694a247f1af35" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f6b93a17812478199d694a247f1af35" member_type: VOTER last_known_addr { host: "127.10.198.62" port: 46393 } } peers { permanent_uuid: "f3fc2bc68b2f4752a289c342d2a82bad" member_type: VOTER last_known_addr { host: "127.10.198.61" port: 46873 } } peers { permanent_uuid: "86c5e1afa805401b81a5391886d1f898" member_type: VOTER last_known_addr { host: "127.10.198.60" port: 38695 } } }
I20260925 22:49:14.321523 1091 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2f6b93a17812478199d694a247f1af35 [sys.catalog]: This master's current role is: LEADER
I20260925 22:49:14.322242 1074 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2f6b93a17812478199d694a247f1af35 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "2f6b93a17812478199d694a247f1af35" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f6b93a17812478199d694a247f1af35" member_type: VOTER last_known_addr { host: "127.10.198.62" port: 46393 } } peers { permanent_uuid: "f3fc2bc68b2f4752a289c342d2a82bad" member_type: VOTER last_known_addr { host: "127.10.198.61" port: 46873 } } peers { permanent_uuid: "86c5e1afa805401b81a5391886d1f898" member_type: VOTER last_known_addr { host: "127.10.198.60" port: 38695 } } }
I20260925 22:49:14.322394 1074 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2f6b93a17812478199d694a247f1af35 [sys.catalog]: This master's current role is: LEADER
I20260925 22:49:14.322553 1098 mvcc.cc:205] Tried to move back new op lower bound from 7333382366397284352 to 7333382366158745600. Current Snapshot: MvccSnapshot[applied={T|T < 7333382366397284352}]
I20260925 22:49:14.322793 1060 sys_catalog.cc:455] T 00000000000000000000000000000000 P 86c5e1afa805401b81a5391886d1f898 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "2f6b93a17812478199d694a247f1af35" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f6b93a17812478199d694a247f1af35" member_type: VOTER last_known_addr { host: "127.10.198.62" port: 46393 } } peers { permanent_uuid: "f3fc2bc68b2f4752a289c342d2a82bad" member_type: VOTER last_known_addr { host: "127.10.198.61" port: 46873 } } peers { permanent_uuid: "86c5e1afa805401b81a5391886d1f898" member_type: VOTER last_known_addr { host: "127.10.198.60" port: 38695 } } }
I20260925 22:49:14.322912 1060 sys_catalog.cc:458] T 00000000000000000000000000000000 P 86c5e1afa805401b81a5391886d1f898 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 22:49:14.323899 1046 sys_catalog.cc:455] T 00000000000000000000000000000000 P f3fc2bc68b2f4752a289c342d2a82bad [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "2f6b93a17812478199d694a247f1af35" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f6b93a17812478199d694a247f1af35" member_type: VOTER last_known_addr { host: "127.10.198.62" port: 46393 } } peers { permanent_uuid: "f3fc2bc68b2f4752a289c342d2a82bad" member_type: VOTER last_known_addr { host: "127.10.198.61" port: 46873 } } peers { permanent_uuid: "86c5e1afa805401b81a5391886d1f898" member_type: VOTER last_known_addr { host: "127.10.198.60" port: 38695 } } }
I20260925 22:49:14.324047 1046 sys_catalog.cc:458] T 00000000000000000000000000000000 P f3fc2bc68b2f4752a289c342d2a82bad [sys.catalog]: This master's current role is: FOLLOWER
I20260925 22:49:14.326547 1093 catalog_manager.cc:1347] Generated new cluster ID: ce659047e3fa492b938d5e10b616c3ef
I20260925 22:49:14.326653 1093 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 22:49:14.356840 1093 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 22:49:14.358640 1093 catalog_manager.cc:1504] Loading token signing keys...
I20260925 22:49:14.371253 1093 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2f6b93a17812478199d694a247f1af35: Generated new TSK 0
I20260925 22:49:14.372309 1093 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260925 22:49:14.397310 893 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:49:14.395553 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41890 (local address 127.10.198.62:46393)
0925 22:49:14.395701 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0925 22:49:14.395706 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:49:14.396048 (+ 342us) server_negotiation.cc:407] Connection header received
0925 22:49:14.396272 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:49:14.396276 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:49:14.396339 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:49:14.396400 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:49:14.397151 (+ 751us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41890: BlockingRecv error: recv got EOF from 127.0.0.1:41890 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
W20260925 22:49:14.397431 966 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:49:14.396363 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54438 (local address 127.10.198.61:46873)
0925 22:49:14.396490 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0925 22:49:14.396495 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:49:14.396516 (+ 21us) server_negotiation.cc:407] Connection header received
0925 22:49:14.396660 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:49:14.396666 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:49:14.396724 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:49:14.396783 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:49:14.397313 (+ 530us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54438: BlockingRecv error: recv got EOF from 127.0.0.1:54438 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
W20260925 22:49:14.397836 1044 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:49:14.396123 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49424 (local address 127.10.198.60:38695)
0925 22:49:14.396259 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0925 22:49:14.396263 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:49:14.396817 (+ 554us) server_negotiation.cc:407] Connection header received
0925 22:49:14.397005 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:49:14.397009 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:49:14.397061 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:49:14.397114 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:49:14.397688 (+ 574us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49424: BlockingRecv error: recv got EOF from 127.0.0.1:49424 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
W20260925 22:49:14.401729 966 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:49:14.400349 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54452 (local address 127.10.198.61:46873)
0925 22:49:14.400478 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0925 22:49:14.400482 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:49:14.400593 (+ 111us) server_negotiation.cc:407] Connection header received
0925 22:49:14.400815 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:49:14.400818 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:49:14.400906 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:49:14.400996 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:49:14.401604 (+ 608us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54452: BlockingRecv error: recv got EOF from 127.0.0.1:54452 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
W20260925 22:49:14.406412 966 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:49:14.405426 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54468 (local address 127.10.198.61:46873)
0925 22:49:14.405550 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0925 22:49:14.405555 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:49:14.405575 (+ 20us) server_negotiation.cc:407] Connection header received
0925 22:49:14.405624 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:49:14.405627 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:49:14.405670 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:49:14.405740 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:49:14.406284 (+ 544us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54468: BlockingRecv error: recv got EOF from 127.0.0.1:54468 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
W20260925 22:49:14.412088 966 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:49:14.410799 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54478 (local address 127.10.198.61:46873)
0925 22:49:14.410948 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0925 22:49:14.410953 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:49:14.411092 (+ 139us) server_negotiation.cc:407] Connection header received
0925 22:49:14.411209 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:49:14.411213 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:49:14.411279 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:49:14.411387 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:49:14.411955 (+ 568us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54478: BlockingRecv error: recv got EOF from 127.0.0.1:54478 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
W20260925 22:49:14.420145 966 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:49:14.418823 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54492 (local address 127.10.198.61:46873)
0925 22:49:14.419011 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0925 22:49:14.419015 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:49:14.419119 (+ 104us) server_negotiation.cc:407] Connection header received
0925 22:49:14.419273 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:49:14.419276 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:49:14.419324 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:49:14.419398 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:49:14.420011 (+ 613us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54492: BlockingRecv error: recv got EOF from 127.0.0.1:54492 (error 108)
Metrics: {"server-negotiator.queue_time_us":100}
W20260925 22:49:14.431797 966 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:49:14.430594 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54496 (local address 127.10.198.61:46873)
0925 22:49:14.430766 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0925 22:49:14.430773 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 22:49:14.430919 (+ 146us) server_negotiation.cc:407] Connection header received
0925 22:49:14.431030 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:49:14.431033 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:49:14.431099 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:49:14.431203 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:49:14.431667 (+ 464us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54496: BlockingRecv error: recv got EOF from 127.0.0.1:54496 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
W20260925 22:49:14.451166 966 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:49:14.450066 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54500 (local address 127.10.198.61:46873)
0925 22:49:14.450209 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0925 22:49:14.450213 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:49:14.450372 (+ 159us) server_negotiation.cc:407] Connection header received
0925 22:49:14.450473 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:49:14.450475 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:49:14.450523 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:49:14.450609 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:49:14.451059 (+ 450us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54500: BlockingRecv error: recv got EOF from 127.0.0.1:54500 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
W20260925 22:49:14.487114 966 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:49:14.485717 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54506 (local address 127.10.198.61:46873)
0925 22:49:14.485886 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0925 22:49:14.485890 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:49:14.486040 (+ 150us) server_negotiation.cc:407] Connection header received
0925 22:49:14.486230 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:49:14.486234 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:49:14.486288 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:49:14.486374 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:49:14.486993 (+ 619us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54506: BlockingRecv error: recv got EOF from 127.0.0.1:54506 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
W20260925 22:49:14.555272 966 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:49:14.553683 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54514 (local address 127.10.198.61:46873)
0925 22:49:14.554003 (+ 320us) server_negotiation.cc:206] Beginning negotiation
0925 22:49:14.554010 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 22:49:14.554334 (+ 324us) server_negotiation.cc:407] Connection header received
0925 22:49:14.554474 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:49:14.554476 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:49:14.554527 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:49:14.554612 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:49:14.555114 (+ 502us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54514: BlockingRecv error: recv got EOF from 127.0.0.1:54514 (error 108)
Metrics: {"server-negotiator.queue_time_us":102}
W20260925 22:49:14.687125 966 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:49:14.685635 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54520 (local address 127.10.198.61:46873)
0925 22:49:14.685876 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0925 22:49:14.685882 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 22:49:14.686055 (+ 173us) server_negotiation.cc:407] Connection header received
0925 22:49:14.686262 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:49:14.686265 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:49:14.686318 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:49:14.686424 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:49:14.686973 (+ 549us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54520: BlockingRecv error: recv got EOF from 127.0.0.1:54520 (error 108)
Metrics: {"server-negotiator.queue_time_us":115}
W20260925 22:49:14.946834 966 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:49:14.945587 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54526 (local address 127.10.198.61:46873)
0925 22:49:14.945779 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0925 22:49:14.945787 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 22:49:14.945830 (+ 43us) server_negotiation.cc:407] Connection header received
0925 22:49:14.946029 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:49:14.946032 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:49:14.946080 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:49:14.946177 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:49:14.946672 (+ 495us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54526: BlockingRecv error: recv got EOF from 127.0.0.1:54526 (error 108)
Metrics: {"server-negotiator.queue_time_us":78}
I20260925 22:49:15.081475 1057 catalog_manager.cc:1259] Loaded cluster ID: ce659047e3fa492b938d5e10b616c3ef
I20260925 22:49:15.081573 1057 catalog_manager.cc:1552] T 00000000000000000000000000000000 P f3fc2bc68b2f4752a289c342d2a82bad: loading cluster ID for follower catalog manager: success
I20260925 22:49:15.083554 1057 catalog_manager.cc:1574] T 00000000000000000000000000000000 P f3fc2bc68b2f4752a289c342d2a82bad: acquiring CA information for follower catalog manager: success
I20260925 22:49:15.085016 1057 catalog_manager.cc:1602] T 00000000000000000000000000000000 P f3fc2bc68b2f4752a289c342d2a82bad: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260925 22:49:15.111305 1071 catalog_manager.cc:1259] Loaded cluster ID: ce659047e3fa492b938d5e10b616c3ef
I20260925 22:49:15.111409 1071 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 86c5e1afa805401b81a5391886d1f898: loading cluster ID for follower catalog manager: success
I20260925 22:49:15.113466 1071 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 86c5e1afa805401b81a5391886d1f898: acquiring CA information for follower catalog manager: success
I20260925 22:49:15.114933 1071 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 86c5e1afa805401b81a5391886d1f898: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260925 22:49:15.465583 1170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:15.461249 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54530 (local address 127.10.198.61:46873)
0925 22:49:15.461539 (+ 290us) server_negotiation.cc:206] Beginning negotiation
0925 22:49:15.461544 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:49:15.461667 (+ 123us) server_negotiation.cc:407] Connection header received
0925 22:49:15.461856 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:49:15.461864 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:49:15.462002 (+ 138us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:49:15.462131 (+ 129us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 22:49:15.462406 (+ 275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:49:15.462955 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:49:15.463952 (+ 997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:49:15.464180 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:49:15.464302 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:49:15.464660 (+ 358us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 22:49:15.465128 (+ 468us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 22:49:15.465267 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0925 22:49:15.465326 (+ 59us) server_negotiation.cc:299] Negotiation successful
0925 22:49:15.465393 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":199,"thread_start_us":97,"threads_started":1}
I20260925 22:49:15.474308 1175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:15.469612 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49438 (local address 127.10.198.60:38695)
0925 22:49:15.469913 (+ 301us) server_negotiation.cc:206] Beginning negotiation
0925 22:49:15.469919 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 22:49:15.470167 (+ 248us) server_negotiation.cc:407] Connection header received
0925 22:49:15.470409 (+ 242us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:49:15.470412 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:49:15.470545 (+ 133us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:49:15.470685 (+ 140us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 22:49:15.470962 (+ 277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:49:15.471534 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:49:15.472499 (+ 965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:49:15.472737 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:49:15.472866 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:49:15.473288 (+ 422us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 22:49:15.473761 (+ 473us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 22:49:15.473917 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0925 22:49:15.473972 (+ 55us) server_negotiation.cc:299] Negotiation successful
0925 22:49:15.474042 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":169,"thread_start_us":78,"threads_started":1}
I20260925 22:49:15.482378 1175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:15.478402 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49440 (local address 127.10.198.60:38695)
0925 22:49:15.478542 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0925 22:49:15.478547 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:49:15.478586 (+ 39us) server_negotiation.cc:407] Connection header received
0925 22:49:15.478777 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:49:15.478781 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:49:15.478847 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:49:15.478913 (+ 66us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 22:49:15.479337 (+ 424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:49:15.480142 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:49:15.481136 (+ 994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:49:15.481380 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:49:15.481520 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:49:15.481812 (+ 292us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 22:49:15.481897 (+ 85us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 22:49:15.481986 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0925 22:49:15.482121 (+ 135us) server_negotiation.cc:299] Negotiation successful
0925 22:49:15.482195 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260925 22:49:15.482998 1170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:15.477808 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54532 (local address 127.10.198.61:46873)
0925 22:49:15.477990 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0925 22:49:15.477995 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:49:15.478528 (+ 533us) server_negotiation.cc:407] Connection header received
0925 22:49:15.478722 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:49:15.478725 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:49:15.478782 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:49:15.478852 (+ 70us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 22:49:15.479349 (+ 497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:49:15.479948 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:49:15.481547 (+ 1599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:49:15.481924 (+ 377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:49:15.482007 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:49:15.482445 (+ 438us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 22:49:15.482529 (+ 84us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 22:49:15.482575 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0925 22:49:15.482741 (+ 166us) server_negotiation.cc:299] Negotiation successful
0925 22:49:15.482804 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260925 22:49:15.484057 1183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:15.478074 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41902 (local address 127.10.198.62:46393)
0925 22:49:15.478343 (+ 269us) server_negotiation.cc:206] Beginning negotiation
0925 22:49:15.478348 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:49:15.478364 (+ 16us) server_negotiation.cc:407] Connection header received
0925 22:49:15.478479 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:49:15.478483 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:49:15.478672 (+ 189us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:49:15.478868 (+ 196us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 22:49:15.479136 (+ 268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:49:15.480015 (+ 879us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:49:15.481547 (+ 1532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:49:15.481915 (+ 368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:49:15.482065 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:49:15.482415 (+ 350us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 22:49:15.482812 (+ 397us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 22:49:15.483780 (+ 968us) server_negotiation.cc:1035] Waiting for connection context
0925 22:49:15.483819 (+ 39us) server_negotiation.cc:299] Negotiation successful
0925 22:49:15.483918 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":189,"thread_start_us":79,"threads_started":1}
I20260925 22:49:15.488134 11032 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrWXm8X/build/debug/bin/kudu with pid 821
I20260925 22:49:15.496172 11032 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrWXm8X/build/debug/bin/kudu with pid 896
I20260925 22:49:15.502966 11032 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrWXm8X/build/debug/bin/kudu with pid 970
2026-09-25T22:49:15Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (3383 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790376259650642-11032-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 22:49:15 dist-test-slave-5vdr krb5kdc[1187](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 22:49:15 dist-test-slave-5vdr krb5kdc[1187](info): set up 2 sockets
Sep 25 22:49:15 dist-test-slave-5vdr krb5kdc[1187](info): commencing operation
krb5kdc: starting...
W20260925 22:49:16.572930 11032 mini_kdc.cc:121] Time spent starting KDC: real 1.046s user 0.001s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790376259650642-11032-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790376259650642-11032-0/krb5kdc/test-user.keytab.
Sep 25 22:49:16 dist-test-slave-5vdr krb5kdc[1187](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376556, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T22:49:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T22:49:16Z Disabled control of system clock
WARNING: no policy specified for kudu/127.10.198.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.198.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.198.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790376259650642-11032-0/krb5kdc/kudu_127.10.198.62.keytab.
Entry for principal kudu/127.10.198.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790376259650642-11032-0/krb5kdc/kudu_127.10.198.62.keytab.
I20260925 22:49:16.706746 11032 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrWXm8X/build/debug/bin/kudu
/tmp/dist-test-taskrWXm8X/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790376259650642-11032-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790376259650642-11032-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790376259650642-11032-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790376259650642-11032-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.198.62:43067
--webserver_interface=127.10.198.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790376259650642-11032-0/krb5kdc/kudu_127.10.198.62.keytab
--principal=kudu/127.10.198.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.198.20:33857
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.198.62:43067,127.10.198.61:41425,127.10.198.60:39493
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790376259650642-11032-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790376259650642-11032-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790376259650642-11032-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 22:49:16.825273 1200 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260925 22:49:16.825788 1200 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:49:16.825862 1200 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:49:16.825947 1200 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:49:16.830636 1200 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 22:49:16.830755 1200 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:49:16.830791 1200 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:49:16.830829 1200 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 22:49:16.830884 1200 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 22:49:16.836895 1200 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.198.20:33857
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790376259650642-11032-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790376259650642-11032-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.198.62:43067,127.10.198.61:41425,127.10.198.60:39493
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790376259650642-11032-0/krb5kdc/kudu_127.10.198.62.keytab
--principal=kudu/127.10.198.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.198.62:43067
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790376259650642-11032-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.198.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1200
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790376259650642-11032-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:09:16 UTC on ccycloud.quasar-afmhyw.root.comops.site
build id 4673379
I20260925 22:49:16.838173 1200 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:49:16.839309 1200 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:49:16.849081 1205 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:49:16.849089 1206 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:49:16.849241 1200 server_base.cc:1034] running on GCE node
W20260925 22:49:16.849081 1208 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:49:16.850515 1200 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:49:16.851955 1200 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:49:16.853202 1200 hybrid_clock.cc:648] HybridClock initialized: now 1790376556853183 us; error 65 us; skew 500 ppm
Sep 25 22:49:16 dist-test-slave-5vdr krb5kdc[1187](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376556, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.198.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:49:16.856439 1200 init.cc:377] Logged in from keytab as kudu/127.10.198.62@KRBTEST.COM (short username kudu)
I20260925 22:49:16.857596 1200 webserver.cc:460] Webserver started at http://127.10.198.62:41891/ using document root <none> and password file <none>
I20260925 22:49:16.858264 1200 fs_manager.cc:359] Metadata directory not provided
I20260925 22:49:16.858385 1200 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:49:16.858651 1200 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:49:16.861238 1200 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790376259650642-11032-0/minicluster-data/master-0/data/instance:
uuid: "64617aec2aef48b883194cc873b75a2f"
format_stamp: "Formatted at 2026-09-25 22:49:16 on dist-test-slave-5vdr"
server_key: "d1f09226f146f4b47aafa51a5af278ba"
server_key_iv: "160c334b1d3dc29855b1c69bd6cf3fd9"
server_key_version: "encryptionkey@0"
I20260925 22:49:16.861931 1200 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790376259650642-11032-0/minicluster-data/master-0/wal/instance:
uuid: "64617aec2aef48b883194cc873b75a2f"
format_stamp: "Formatted at 2026-09-25 22:49:16 on dist-test-slave-5vdr"
server_key: "d1f09226f146f4b47aafa51a5af278ba"
server_key_iv: "160c334b1d3dc29855b1c69bd6cf3fd9"
server_key_version: "encryptionkey@0"
I20260925 22:49:16.866525 1200 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260925 22:49:16.869786 1215 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:49:16.871255 1200 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260925 22:49:16.871466 1200 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790376259650642-11032-0/minicluster-data/master-0/data,/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790376259650642-11032-0/minicluster-data/master-0/wal
uuid: "64617aec2aef48b883194cc873b75a2f"
format_stamp: "Formatted at 2026-09-25 22:49:16 on dist-test-slave-5vdr"
server_key: "d1f09226f146f4b47aafa51a5af278ba"
server_key_iv: "160c334b1d3dc29855b1c69bd6cf3fd9"
server_key_version: "encryptionkey@0"
I20260925 22:49:16.871630 1200 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790376259650642-11032-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790376259650642-11032-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790376259650642-11032-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:49:16.884794 1200 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:49:16.888341 1200 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:49:16.888592 1200 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:49:16.897240 1200 rpc_server.cc:307] RPC server started. Bound to: 127.10.198.62:43067
I20260925 22:49:16.897562 1267 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.198.62:43067 every 8 connection(s)
I20260925 22:49:16.898428 1200 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790376259650642-11032-0/minicluster-data/master-0/data/info.pb
I20260925 22:49:16.901665 1268 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:49:16.904289 11032 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrWXm8X/build/debug/bin/kudu as pid 1200
I20260925 22:49:16.904469 11032 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790376259650642-11032-0/minicluster-data/master-0/wal/instance
I20260925 22:49:16.904742 11032 external_mini_cluster.cc:1428] Setting key fbdab80cdb6cde9e50858f3070d85290
I20260925 22:49:16.906239 1268 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.198.62" port: 43067 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.10.198.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.198.61@KRBTEST.COM" created.
Sep 25 22:49:16 dist-test-slave-5vdr krb5kdc[1187](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376556, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.198.62@KRBTEST.COM for kudu/127.10.198.62@KRBTEST.COM
I20260925 22:49:16.925632 1271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:16.909076 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44862 (local address 127.10.198.62:43067)
0925 22:49:16.909641 (+ 565us) server_negotiation.cc:206] Beginning negotiation
0925 22:49:16.909661 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0925 22:49:16.910118 (+ 457us) server_negotiation.cc:407] Connection header received
0925 22:49:16.911500 (+ 1382us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:49:16.911506 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:49:16.911879 (+ 373us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:49:16.912127 (+ 248us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:49:16.913219 (+ 1092us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:49:16.914840 (+ 1621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:49:16.916587 (+ 1747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:49:16.917025 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:49:16.917187 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:49:16.920203 (+ 3016us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:49:16.920211 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:49:16.920232 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:49:16.920255 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:49:16.922064 (+ 1809us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:49:16.922626 (+ 562us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:49:16.922632 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:49:16.922634 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:49:16.922758 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:49:16.923095 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:49:16.923100 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:49:16.923102 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:49:16.923549 (+ 447us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:49:16.923706 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0925 22:49:16.924152 (+ 446us) server_negotiation.cc:299] Negotiation successful
0925 22:49:16.924502 (+ 350us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":44,"server-negotiator.queue_time_us":381,"thread_start_us":190,"threads_started":1}
I20260925 22:49:16.925632 1272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:16.909076 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.198.62:43067 (local address 127.0.0.1:44862)
0925 22:49:16.909916 (+ 840us) negotiation.cc:107] Waiting for socket to connect
0925 22:49:16.909964 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0925 22:49:16.911068 (+ 1104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:49:16.912150 (+ 1082us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:49:16.912157 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:49:16.912722 (+ 565us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:49:16.913019 (+ 297us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:49:16.913037 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:49:16.915013 (+ 1976us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:49:16.915019 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:49:16.916406 (+ 1387us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:49:16.916427 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:49:16.917330 (+ 903us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:49:16.917343 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:49:16.917551 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:49:16.918313 (+ 762us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:49:16.918333 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:49:16.920018 (+ 1685us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:49:16.922212 (+ 2194us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:49:16.922223 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:49:16.922235 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:49:16.922502 (+ 267us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:49:16.922866 (+ 364us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:49:16.922873 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:49:16.922875 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:49:16.922999 (+ 124us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:49:16.923671 (+ 672us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:49:16.923678 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:49:16.923830 (+ 152us) client_negotiation.cc:769] Sending connection context
0925 22:49:16.924090 (+ 260us) client_negotiation.cc:240] Negotiation successful
0925 22:49:16.924554 (+ 464us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":666,"mutex_wait_us":120,"spinlock_wait_cycles":4864,"thread_start_us":203,"threads_started":1}
I20260925 22:49:16.928558 1268 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.198.61" port: 41425 } has no permanent_uuid. Determining permanent_uuid...
I20260925 22:49:16.929507 1272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:16.929151 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.198.61:41425 (local address 127.0.0.1:39856)
0925 22:49:16.929343 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0925 22:49:16.929428 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.198.61:41425: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":91}
W20260925 22:49:16.929767 1218 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.10.198.61:41425: connect: Connection refused (error 111)
I20260925 22:49:16.930855 1272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:16.930599 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.198.61:41425 (local address 127.0.0.1:39866)
0925 22:49:16.930722 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0925 22:49:16.930781 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.198.61:41425: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260925 22:49:16.931077 1268 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.10.198.61:41425: Network error: Client connection negotiation failed: client connection to 127.10.198.61:41425: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.198.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790376259650642-11032-0/krb5kdc/kudu_127.10.198.61.keytab.
Entry for principal kudu/127.10.198.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790376259650642-11032-0/krb5kdc/kudu_127.10.198.61.keytab.
I20260925 22:49:16.934777 11032 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrWXm8X/build/debug/bin/kudu
/tmp/dist-test-taskrWXm8X/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790376259650642-11032-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790376259650642-11032-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790376259650642-11032-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790376259650642-11032-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.198.61:41425
--webserver_interface=127.10.198.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790376259650642-11032-0/krb5kdc/kudu_127.10.198.61.keytab
--principal=kudu/127.10.198.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.198.20:33857
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.198.62:43067,127.10.198.61:41425,127.10.198.60:39493
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790376259650642-11032-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790376259650642-11032-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790376259650642-11032-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 22:49:16.980377 1268 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.198.61" port: 41425 } attempt: 1
I20260925 22:49:16.981356 1272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:16.981024 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.198.61:41425 (local address 127.0.0.1:39876)
0925 22:49:16.981193 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0925 22:49:16.981275 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.198.61:41425: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
I20260925 22:49:16.982326 1272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:16.982042 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.198.61:41425 (local address 127.0.0.1:39886)
0925 22:49:16.982194 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0925 22:49:16.982260 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.198.61:41425: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
W20260925 22:49:16.982595 1268 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.10.198.61:41425: Network error: Client connection negotiation failed: client connection to 127.10.198.61:41425: connect: Connection refused (error 111)
I20260925 22:49:17.050827 1268 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.198.61" port: 41425 } attempt: 2
I20260925 22:49:17.051699 1272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:17.051393 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.198.61:41425 (local address 127.0.0.1:39894)
0925 22:49:17.051549 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0925 22:49:17.051621 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.198.61:41425: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
I20260925 22:49:17.052649 1272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:17.052390 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.198.61:41425 (local address 127.0.0.1:39908)
0925 22:49:17.052520 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0925 22:49:17.052583 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.198.61:41425: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260925 22:49:17.052873 1268 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.10.198.61:41425: Network error: Client connection negotiation failed: client connection to 127.10.198.61:41425: connect: Connection refused (error 111)
W20260925 22:49:17.055702 1275 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260925 22:49:17.056151 1275 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:49:17.056231 1275 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:49:17.056339 1275 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 22:49:17.060881 1275 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 22:49:17.061013 1275 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:49:17.061049 1275 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:49:17.061080 1275 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 22:49:17.061120 1275 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 22:49:17.067950 1275 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.198.20:33857
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790376259650642-11032-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790376259650642-11032-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.10.198.62:43067,127.10.198.61:41425,127.10.198.60:39493
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790376259650642-11032-0/krb5kdc/kudu_127.10.198.61.keytab
--principal=kudu/127.10.198.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.198.61:41425
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790376259650642-11032-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.10.198.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1275
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790376259650642-11032-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:09:16 UTC on ccycloud.quasar-afmhyw.root.comops.site
build id 4673379
I20260925 22:49:17.069274 1275 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:49:17.070453 1275 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:49:17.080735 1281 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:49:17.080754 1283 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:49:17.080758 1280 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:49:17.081056 1275 server_base.cc:1034] running on GCE node
I20260925 22:49:17.081852 1275 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:49:17.083165 1275 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:49:17.084388 1275 hybrid_clock.cc:648] HybridClock initialized: now 1790376557084372 us; error 37 us; skew 500 ppm
Sep 25 22:49:17 dist-test-slave-5vdr krb5kdc[1187](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376557, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.198.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:49:17.088111 1275 init.cc:377] Logged in from keytab as kudu/127.10.198.61@KRBTEST.COM (short username kudu)
I20260925 22:49:17.089352 1275 webserver.cc:460] Webserver started at http://127.10.198.61:43927/ using document root <none> and password file <none>
I20260925 22:49:17.090081 1275 fs_manager.cc:359] Metadata directory not provided
I20260925 22:49:17.090202 1275 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:49:17.090740 1275 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:49:17.093500 1275 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790376259650642-11032-0/minicluster-data/master-1/data/instance:
uuid: "6206349d187a4ed491c2b1822bb6cd16"
format_stamp: "Formatted at 2026-09-25 22:49:17 on dist-test-slave-5vdr"
server_key: "8780e60c95c4cb2220a033a8e2387550"
server_key_iv: "82ebdd481cbac8a35bf5b664b24cc442"
server_key_version: "encryptionkey@0"
I20260925 22:49:17.094225 1275 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790376259650642-11032-0/minicluster-data/master-1/wal/instance:
uuid: "6206349d187a4ed491c2b1822bb6cd16"
format_stamp: "Formatted at 2026-09-25 22:49:17 on dist-test-slave-5vdr"
server_key: "8780e60c95c4cb2220a033a8e2387550"
server_key_iv: "82ebdd481cbac8a35bf5b664b24cc442"
server_key_version: "encryptionkey@0"
I20260925 22:49:17.098805 1275 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260925 22:49:17.101889 1290 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:49:17.103399 1275 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260925 22:49:17.103622 1275 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790376259650642-11032-0/minicluster-data/master-1/data,/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790376259650642-11032-0/minicluster-data/master-1/wal
uuid: "6206349d187a4ed491c2b1822bb6cd16"
format_stamp: "Formatted at 2026-09-25 22:49:17 on dist-test-slave-5vdr"
server_key: "8780e60c95c4cb2220a033a8e2387550"
server_key_iv: "82ebdd481cbac8a35bf5b664b24cc442"
server_key_version: "encryptionkey@0"
I20260925 22:49:17.103776 1275 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790376259650642-11032-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790376259650642-11032-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790376259650642-11032-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:49:17.121881 1275 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:49:17.125942 1275 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:49:17.126220 1275 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:49:17.134491 1275 rpc_server.cc:307] RPC server started. Bound to: 127.10.198.61:41425
I20260925 22:49:17.134848 1342 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.198.61:41425 every 8 connection(s)
I20260925 22:49:17.135689 1275 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790376259650642-11032-0/minicluster-data/master-1/data/info.pb
I20260925 22:49:17.138936 1343 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:49:17.142879 11032 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrWXm8X/build/debug/bin/kudu as pid 1275
I20260925 22:49:17.143090 11032 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790376259650642-11032-0/minicluster-data/master-1/wal/instance
I20260925 22:49:17.143394 11032 external_mini_cluster.cc:1428] Setting key adaacc26bfeee1080a8a1982c8125f7a
I20260925 22:49:17.143523 1343 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.198.62" port: 43067 } has no permanent_uuid. Determining permanent_uuid...
I20260925 22:49:17.144065 1268 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.198.61" port: 41425 } attempt: 3
WARNING: no policy specified for kudu/127.10.198.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.198.60@KRBTEST.COM" created.
Sep 25 22:49:17 dist-test-slave-5vdr krb5kdc[1187](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376557, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.198.61@KRBTEST.COM for kudu/127.10.198.62@KRBTEST.COM
Sep 25 22:49:17 dist-test-slave-5vdr krb5kdc[1187](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376556, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.198.62@KRBTEST.COM for kudu/127.10.198.61@KRBTEST.COM
I20260925 22:49:17.163889 1272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:17.144621 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.198.61:41425 (local address 127.0.0.1:39914)
0925 22:49:17.144842 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0925 22:49:17.144860 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0925 22:49:17.144974 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:49:17.148168 (+ 3194us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:49:17.148172 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:49:17.148470 (+ 298us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:49:17.148644 (+ 174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:49:17.148655 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:49:17.150795 (+ 2140us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:49:17.150798 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:49:17.151532 (+ 734us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:49:17.151544 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:49:17.152401 (+ 857us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:49:17.152406 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:49:17.152549 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:49:17.153169 (+ 620us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:49:17.153185 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:49:17.157916 (+ 4731us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:49:17.161136 (+ 3220us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:49:17.161143 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:49:17.161148 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:49:17.161474 (+ 326us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:49:17.161915 (+ 441us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:49:17.161919 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:49:17.161921 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:49:17.162013 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:49:17.162876 (+ 863us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:49:17.162879 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:49:17.162984 (+ 105us) client_negotiation.cc:769] Sending connection context
0925 22:49:17.163095 (+ 111us) client_negotiation.cc:240] Negotiation successful
0925 22:49:17.163624 (+ 529us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135}
I20260925 22:49:17.164188 1271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:17.146211 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44878 (local address 127.10.198.62:43067)
0925 22:49:17.146355 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0925 22:49:17.146360 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:49:17.146750 (+ 390us) server_negotiation.cc:407] Connection header received
0925 22:49:17.147879 (+ 1129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:49:17.147887 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:49:17.147955 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:49:17.148054 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:49:17.149220 (+ 1166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:49:17.149764 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:49:17.151349 (+ 1585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:49:17.151787 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:49:17.151981 (+ 194us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:49:17.157950 (+ 5969us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:49:17.157958 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:49:17.157964 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:49:17.158012 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:49:17.161008 (+ 2996us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:49:17.161717 (+ 709us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:49:17.161725 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:49:17.161728 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:49:17.161837 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:49:17.162296 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:49:17.162299 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:49:17.162301 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:49:17.162521 (+ 220us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:49:17.162615 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0925 22:49:17.163265 (+ 650us) server_negotiation.cc:299] Negotiation successful
0925 22:49:17.164026 (+ 761us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":555,"server-negotiator.queue_time_us":72}
I20260925 22:49:17.165058 1346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:17.145553 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39914 (local address 127.10.198.61:41425)
0925 22:49:17.146156 (+ 603us) server_negotiation.cc:206] Beginning negotiation
0925 22:49:17.146184 (+ 28us) server_negotiation.cc:399] Waiting for connection header
0925 22:49:17.146240 (+ 56us) server_negotiation.cc:407] Connection header received
0925 22:49:17.147552 (+ 1312us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:49:17.147586 (+ 34us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:49:17.147942 (+ 356us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:49:17.148838 (+ 896us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:49:17.148992 (+ 154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:49:17.150607 (+ 1615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:49:17.151666 (+ 1059us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:49:17.152243 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:49:17.152367 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:49:17.158093 (+ 5726us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:49:17.158106 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:49:17.158129 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:49:17.158174 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:49:17.160939 (+ 2765us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:49:17.161623 (+ 684us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:49:17.161633 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:49:17.161635 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:49:17.161797 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:49:17.162124 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:49:17.162128 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:49:17.162129 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:49:17.162755 (+ 626us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:49:17.162898 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0925 22:49:17.163210 (+ 312us) server_negotiation.cc:299] Negotiation successful
0925 22:49:17.163901 (+ 691us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":30,"server-negotiator.queue_time_us":413,"thread_start_us":163,"threads_started":1}
I20260925 22:49:17.165066 1347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:17.146023 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.198.62:43067 (local address 127.0.0.1:44878)
0925 22:49:17.146602 (+ 579us) negotiation.cc:107] Waiting for socket to connect
0925 22:49:17.146656 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0925 22:49:17.147569 (+ 913us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:49:17.148120 (+ 551us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:49:17.148132 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:49:17.148534 (+ 402us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:49:17.149058 (+ 524us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:49:17.149069 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:49:17.149902 (+ 833us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:49:17.149906 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:49:17.151178 (+ 1272us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:49:17.151190 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:49:17.152020 (+ 830us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:49:17.152023 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:49:17.152217 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:49:17.152976 (+ 759us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:49:17.153016 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:49:17.157685 (+ 4669us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:49:17.161153 (+ 3468us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:49:17.161165 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:49:17.161180 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:49:17.161525 (+ 345us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:49:17.161988 (+ 463us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:49:17.161995 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:49:17.161999 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:49:17.162180 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:49:17.162675 (+ 495us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:49:17.162685 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:49:17.162909 (+ 224us) client_negotiation.cc:769] Sending connection context
0925 22:49:17.163369 (+ 460us) client_negotiation.cc:240] Negotiation successful
0925 22:49:17.164417 (+ 1048us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":408,"mutex_wait_us":758,"thread_start_us":117,"threads_started":1}
I20260925 22:49:17.166802 1343 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.198.61" port: 41425 } has no permanent_uuid. Determining permanent_uuid...
I20260925 22:49:17.167923 1268 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.198.60" port: 39493 } has no permanent_uuid. Determining permanent_uuid...
I20260925 22:49:17.168850 1272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:17.168472 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.198.60:39493 (local address 127.0.0.1:44980)
0925 22:49:17.168599 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0925 22:49:17.168658 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.198.60:39493: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260925 22:49:17.169736 1272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:17.169372 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.198.60:39493 (local address 127.0.0.1:44992)
0925 22:49:17.169570 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0925 22:49:17.169642 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.198.60:39493: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103}
W20260925 22:49:17.169951 1268 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.10.198.60:39493: Network error: Client connection negotiation failed: client connection to 127.10.198.60:39493: connect: Connection refused (error 111)
Sep 25 22:49:17 dist-test-slave-5vdr krb5kdc[1187](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376557, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.198.61@KRBTEST.COM for kudu/127.10.198.61@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.198.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790376259650642-11032-0/krb5kdc/kudu_127.10.198.60.keytab.
Entry for principal kudu/127.10.198.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790376259650642-11032-0/krb5kdc/kudu_127.10.198.60.keytab.
I20260925 22:49:17.175472 11032 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrWXm8X/build/debug/bin/kudu
/tmp/dist-test-taskrWXm8X/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790376259650642-11032-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790376259650642-11032-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790376259650642-11032-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790376259650642-11032-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.198.60:39493
--webserver_interface=127.10.198.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790376259650642-11032-0/krb5kdc/kudu_127.10.198.60.keytab
--principal=kudu/127.10.198.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.198.20:33857
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.198.62:43067,127.10.198.61:41425,127.10.198.60:39493
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790376259650642-11032-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790376259650642-11032-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790376259650642-11032-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 22:49:17.179319 1347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:17.167589 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.198.61:41425 (local address 127.0.0.1:39928)
0925 22:49:17.167814 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0925 22:49:17.167834 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0925 22:49:17.167968 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:49:17.168260 (+ 292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:49:17.168263 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:49:17.168470 (+ 207us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:49:17.168568 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:49:17.168575 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:49:17.169645 (+ 1070us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:49:17.169652 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:49:17.170573 (+ 921us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:49:17.170587 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:49:17.171142 (+ 555us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:49:17.171145 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:49:17.171239 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:49:17.171776 (+ 537us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:49:17.171793 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:49:17.173506 (+ 1713us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:49:17.177622 (+ 4116us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:49:17.177627 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:49:17.177630 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:49:17.177860 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:49:17.178177 (+ 317us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:49:17.178180 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:49:17.178182 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:49:17.178260 (+ 78us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:49:17.178678 (+ 418us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:49:17.178681 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:49:17.178774 (+ 93us) client_negotiation.cc:769] Sending connection context
0925 22:49:17.178887 (+ 113us) client_negotiation.cc:240] Negotiation successful
0925 22:49:17.179162 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":91,"mutex_wait_us":123}
I20260925 22:49:17.179319 1346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:17.167817 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39928 (local address 127.10.198.61:41425)
0925 22:49:17.167951 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0925 22:49:17.167955 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:49:17.167970 (+ 15us) server_negotiation.cc:407] Connection header received
0925 22:49:17.168079 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:49:17.168085 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:49:17.168138 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:49:17.168250 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:49:17.168701 (+ 451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:49:17.169517 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:49:17.170731 (+ 1214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:49:17.171013 (+ 282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:49:17.171143 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:49:17.173706 (+ 2563us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:49:17.173711 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:49:17.173713 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:49:17.173739 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:49:17.177450 (+ 3711us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:49:17.177982 (+ 532us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:49:17.177988 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:49:17.177990 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:49:17.178073 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:49:17.178378 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:49:17.178381 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:49:17.178383 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:49:17.178581 (+ 198us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:49:17.178692 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0925 22:49:17.178931 (+ 239us) server_negotiation.cc:299] Negotiation successful
0925 22:49:17.179106 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":4,"server-negotiator.queue_time_us":69}
I20260925 22:49:17.180250 1343 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.198.60" port: 39493 } has no permanent_uuid. Determining permanent_uuid...
I20260925 22:49:17.181253 1347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:17.180848 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.198.60:39493 (local address 127.0.0.1:44998)
0925 22:49:17.181025 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0925 22:49:17.181145 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.198.60:39493: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
W20260925 22:49:17.181523 1293 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.10.198.60:39493: connect: Connection refused (error 111)
I20260925 22:49:17.182554 1347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:17.182230 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.198.60:39493 (local address 127.0.0.1:45010)
0925 22:49:17.182378 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0925 22:49:17.182450 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.198.60:39493: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260925 22:49:17.182791 1343 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.10.198.60:39493: Network error: Client connection negotiation failed: client connection to 127.10.198.60:39493: connect: Connection refused (error 111)
I20260925 22:49:17.201184 1268 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.198.60" port: 39493 } attempt: 1
I20260925 22:49:17.202219 1272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:17.201878 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.198.60:39493 (local address 127.0.0.1:45012)
0925 22:49:17.202052 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0925 22:49:17.202124 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.198.60:39493: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
I20260925 22:49:17.203495 1272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:17.202968 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.198.60:39493 (local address 127.0.0.1:45014)
0925 22:49:17.203240 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0925 22:49:17.203359 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.198.60:39493: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113}
W20260925 22:49:17.203943 1268 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.10.198.60:39493: Network error: Client connection negotiation failed: client connection to 127.10.198.60:39493: connect: Connection refused (error 111)
I20260925 22:49:17.232129 1343 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.198.60" port: 39493 } attempt: 1
I20260925 22:49:17.233130 1347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:17.232741 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.198.60:39493 (local address 127.0.0.1:45016)
0925 22:49:17.232954 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0925 22:49:17.233032 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.198.60:39493: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":92}
I20260925 22:49:17.234009 1347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:17.233722 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.198.60:39493 (local address 127.0.0.1:45018)
0925 22:49:17.233867 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0925 22:49:17.233928 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.198.60:39493: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260925 22:49:17.234228 1343 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.10.198.60:39493: Network error: Client connection negotiation failed: client connection to 127.10.198.60:39493: connect: Connection refused (error 111)
I20260925 22:49:17.279220 1268 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.198.60" port: 39493 } attempt: 2
I20260925 22:49:17.280306 1272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:17.279940 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.198.60:39493 (local address 127.0.0.1:45024)
0925 22:49:17.280119 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0925 22:49:17.280203 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.198.60:39493: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
I20260925 22:49:17.281315 1272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:17.281056 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.198.60:39493 (local address 127.0.0.1:45040)
0925 22:49:17.281175 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0925 22:49:17.281244 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.198.60:39493: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260925 22:49:17.281466 1268 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.10.198.60:39493: Network error: Client connection negotiation failed: client connection to 127.10.198.60:39493: connect: Connection refused (error 111)
W20260925 22:49:17.301164 1349 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260925 22:49:17.301584 1349 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 22:49:17.301641 1349 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 22:49:17.301721 1349 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260925 22:49:17.302439 1343 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.198.60" port: 39493 } attempt: 2
I20260925 22:49:17.303401 1347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:17.303037 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.198.60:39493 (local address 127.0.0.1:45046)
0925 22:49:17.303221 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0925 22:49:17.303302 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.198.60:39493: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
I20260925 22:49:17.304328 1347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:17.304067 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.198.60:39493 (local address 127.0.0.1:45058)
0925 22:49:17.304193 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0925 22:49:17.304252 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.198.60:39493: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260925 22:49:17.304536 1343 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.10.198.60:39493: Network error: Client connection negotiation failed: client connection to 127.10.198.60:39493: connect: Connection refused (error 111)
W20260925 22:49:17.306855 1349 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 22:49:17.306952 1349 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 22:49:17.306973 1349 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 22:49:17.306993 1349 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 22:49:17.307009 1349 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 22:49:17.313767 1349 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.198.20:33857
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790376259650642-11032-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790376259650642-11032-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.10.198.62:43067,127.10.198.61:41425,127.10.198.60:39493
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790376259650642-11032-0/krb5kdc/kudu_127.10.198.60.keytab
--principal=kudu/127.10.198.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.198.60:39493
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790376259650642-11032-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.10.198.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1349
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790376259650642-11032-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 25 Sep 2026 22:09:16 UTC on ccycloud.quasar-afmhyw.root.comops.site
build id 4673379
I20260925 22:49:17.315059 1349 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 22:49:17.316318 1349 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 22:49:17.325907 1356 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:49:17.325845 1358 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 22:49:17.325922 1355 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 22:49:17.326100 1349 server_base.cc:1034] running on GCE node
I20260925 22:49:17.327085 1349 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 22:49:17.328620 1349 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 22:49:17.329861 1349 hybrid_clock.cc:648] HybridClock initialized: now 1790376557329843 us; error 49 us; skew 500 ppm
Sep 25 22:49:17 dist-test-slave-5vdr krb5kdc[1187](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376557, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.198.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 22:49:17.333241 1349 init.cc:377] Logged in from keytab as kudu/127.10.198.60@KRBTEST.COM (short username kudu)
I20260925 22:49:17.334336 1349 webserver.cc:460] Webserver started at http://127.10.198.60:42073/ using document root <none> and password file <none>
I20260925 22:49:17.334991 1349 fs_manager.cc:359] Metadata directory not provided
I20260925 22:49:17.335103 1349 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 22:49:17.335368 1349 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 22:49:17.337775 1349 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790376259650642-11032-0/minicluster-data/master-2/data/instance:
uuid: "00ac2a44184548daa6c6ecd21ddacbe4"
format_stamp: "Formatted at 2026-09-25 22:49:17 on dist-test-slave-5vdr"
server_key: "1ee6a997dac9265da5d1d9de36512c40"
server_key_iv: "b164d8f02556ff5dc65530367c4621b6"
server_key_version: "encryptionkey@0"
I20260925 22:49:17.338457 1349 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790376259650642-11032-0/minicluster-data/master-2/wal/instance:
uuid: "00ac2a44184548daa6c6ecd21ddacbe4"
format_stamp: "Formatted at 2026-09-25 22:49:17 on dist-test-slave-5vdr"
server_key: "1ee6a997dac9265da5d1d9de36512c40"
server_key_iv: "b164d8f02556ff5dc65530367c4621b6"
server_key_version: "encryptionkey@0"
I20260925 22:49:17.343080 1349 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.001s
I20260925 22:49:17.346310 1365 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 22:49:17.347808 1349 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260925 22:49:17.348100 1349 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790376259650642-11032-0/minicluster-data/master-2/data,/tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790376259650642-11032-0/minicluster-data/master-2/wal
uuid: "00ac2a44184548daa6c6ecd21ddacbe4"
format_stamp: "Formatted at 2026-09-25 22:49:17 on dist-test-slave-5vdr"
server_key: "1ee6a997dac9265da5d1d9de36512c40"
server_key_iv: "b164d8f02556ff5dc65530367c4621b6"
server_key_version: "encryptionkey@0"
I20260925 22:49:17.348248 1349 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790376259650642-11032-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790376259650642-11032-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790376259650642-11032-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 22:49:17.365088 1349 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 22:49:17.368505 1349 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 22:49:17.368741 1349 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 22:49:17.377111 1349 rpc_server.cc:307] RPC server started. Bound to: 127.10.198.60:39493
I20260925 22:49:17.377446 1417 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.198.60:39493 every 8 connection(s)
I20260925 22:49:17.378366 1349 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790376259650642-11032-0/minicluster-data/master-2/data/info.pb
I20260925 22:49:17.380806 1268 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.198.60" port: 39493 } attempt: 3
I20260925 22:49:17.382192 1418 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 22:49:17.383544 11032 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrWXm8X/build/debug/bin/kudu as pid 1349
I20260925 22:49:17.383698 11032 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrWXm8X/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790376259650642-11032-0/minicluster-data/master-2/wal/instance
I20260925 22:49:17.384038 11032 external_mini_cluster.cc:1428] Setting key 34cc83bdf0e30c778ffbf3f41c7b066a
I20260925 22:49:17.387866 1418 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.198.62" port: 43067 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 22:49:17 dist-test-slave-5vdr krb5kdc[1187](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376556, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.198.62@KRBTEST.COM for kudu/127.10.198.60@KRBTEST.COM
Sep 25 22:49:17 dist-test-slave-5vdr krb5kdc[1187](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376556, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.198.60@KRBTEST.COM
I20260925 22:49:17.395731 1343 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.198.60" port: 39493 } attempt: 3
I20260925 22:49:17.397315 1272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:17.381329 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.198.60:39493 (local address 127.0.0.1:45068)
0925 22:49:17.381833 (+ 504us) negotiation.cc:107] Waiting for socket to connect
0925 22:49:17.381850 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 22:49:17.381953 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:49:17.385470 (+ 3517us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:49:17.385475 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:49:17.385711 (+ 236us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:49:17.385833 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:49:17.385841 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:49:17.387690 (+ 1849us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:49:17.387694 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:49:17.388536 (+ 842us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:49:17.388550 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:49:17.389227 (+ 677us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:49:17.389231 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:49:17.389318 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:49:17.389936 (+ 618us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:49:17.389948 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:49:17.392195 (+ 2247us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:49:17.394964 (+ 2769us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:49:17.394968 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:49:17.394970 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:49:17.395202 (+ 232us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:49:17.395726 (+ 524us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:49:17.395729 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:49:17.395731 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:49:17.395831 (+ 100us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:49:17.396758 (+ 927us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:49:17.396762 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:49:17.396875 (+ 113us) client_negotiation.cc:769] Sending connection context
0925 22:49:17.396972 (+ 97us) client_negotiation.cc:240] Negotiation successful
0925 22:49:17.397127 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":415}
Sep 25 22:49:17 dist-test-slave-5vdr krb5kdc[1187](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376557, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.198.60@KRBTEST.COM for kudu/127.10.198.62@KRBTEST.COM
I20260925 22:49:17.399459 1420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:17.382703 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45068 (local address 127.10.198.60:39493)
0925 22:49:17.383314 (+ 611us) server_negotiation.cc:206] Beginning negotiation
0925 22:49:17.383336 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0925 22:49:17.383384 (+ 48us) server_negotiation.cc:407] Connection header received
0925 22:49:17.384512 (+ 1128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:49:17.384539 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:49:17.385049 (+ 510us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:49:17.385524 (+ 475us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:49:17.386032 (+ 508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:49:17.387522 (+ 1490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:49:17.388698 (+ 1176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:49:17.389085 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:49:17.389576 (+ 491us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:49:17.392386 (+ 2810us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:49:17.392394 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:49:17.392407 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:49:17.392441 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:49:17.394819 (+ 2378us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:49:17.395360 (+ 541us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:49:17.395369 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:49:17.395378 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:49:17.395549 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:49:17.395974 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:49:17.395977 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:49:17.395979 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:49:17.396629 (+ 650us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:49:17.397606 (+ 977us) server_negotiation.cc:1035] Waiting for connection context
0925 22:49:17.397798 (+ 192us) server_negotiation.cc:299] Negotiation successful
0925 22:49:17.398221 (+ 423us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":140,"server-negotiator.queue_time_us":447,"spinlock_wait_cycles":2432,"thread_start_us":157,"threads_started":1}
I20260925 22:49:17.399471 1421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:17.384700 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45074 (local address 127.10.198.60:39493)
0925 22:49:17.385027 (+ 327us) server_negotiation.cc:206] Beginning negotiation
0925 22:49:17.385035 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 22:49:17.385295 (+ 260us) server_negotiation.cc:407] Connection header received
0925 22:49:17.385514 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:49:17.385520 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:49:17.385579 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:49:17.385665 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:49:17.386734 (+ 1069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:49:17.387520 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:49:17.388516 (+ 996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:49:17.389050 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:49:17.389229 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:49:17.392461 (+ 3232us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:49:17.392464 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:49:17.392466 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:49:17.392489 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:49:17.394876 (+ 2387us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:49:17.395454 (+ 578us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:49:17.395457 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:49:17.395459 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:49:17.395576 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:49:17.395962 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:49:17.395966 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:49:17.395968 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:49:17.396630 (+ 662us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:49:17.396840 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0925 22:49:17.397801 (+ 961us) server_negotiation.cc:299] Negotiation successful
0925 22:49:17.398318 (+ 517us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":315,"server-negotiator.queue_time_us":253,"spinlock_wait_cycles":18688,"thread_start_us":105,"threads_started":1}
Sep 25 22:49:17 dist-test-slave-5vdr krb5kdc[1187](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376557, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.198.61@KRBTEST.COM for kudu/127.10.198.60@KRBTEST.COM
I20260925 22:49:17.403040 1271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:17.390996 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44894 (local address 127.10.198.62:43067)
0925 22:49:17.391118 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0925 22:49:17.391122 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:49:17.391137 (+ 15us) server_negotiation.cc:407] Connection header received
0925 22:49:17.391215 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:49:17.391217 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:49:17.391262 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:49:17.391352 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:49:17.392224 (+ 872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:49:17.392986 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:49:17.394259 (+ 1273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:49:17.394653 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:49:17.394767 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:49:17.398694 (+ 3927us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:49:17.398698 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:49:17.398701 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:49:17.398734 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:49:17.400747 (+ 2013us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:49:17.401395 (+ 648us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:49:17.401399 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:49:17.401401 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:49:17.401497 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:49:17.401814 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:49:17.401817 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:49:17.401819 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:49:17.402145 (+ 326us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:49:17.402224 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0925 22:49:17.402719 (+ 495us) server_negotiation.cc:299] Negotiation successful
0925 22:49:17.402866 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260925 22:49:17.403044 1423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:17.390081 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.198.62:43067 (local address 127.0.0.1:44894)
0925 22:49:17.390923 (+ 842us) negotiation.cc:107] Waiting for socket to connect
0925 22:49:17.390985 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0925 22:49:17.391121 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:49:17.391436 (+ 315us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:49:17.391443 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:49:17.391964 (+ 521us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:49:17.392092 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:49:17.392098 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:49:17.393105 (+ 1007us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:49:17.393110 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:49:17.394084 (+ 974us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:49:17.394102 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:49:17.394958 (+ 856us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:49:17.394972 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:49:17.395116 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:49:17.395677 (+ 561us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:49:17.395691 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:49:17.398509 (+ 2818us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:49:17.400897 (+ 2388us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:49:17.400906 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:49:17.400912 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:49:17.401268 (+ 356us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:49:17.401620 (+ 352us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:49:17.401623 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:49:17.401625 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:49:17.401711 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:49:17.402363 (+ 652us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:49:17.402369 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:49:17.402491 (+ 122us) client_negotiation.cc:769] Sending connection context
0925 22:49:17.402662 (+ 171us) client_negotiation.cc:240] Negotiation successful
0925 22:49:17.402866 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":623,"thread_start_us":166,"threads_started":1}
I20260925 22:49:17.404526 1418 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.198.61" port: 41425 } has no permanent_uuid. Determining permanent_uuid...
I20260925 22:49:17.405321 1268 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 64617aec2aef48b883194cc873b75a2f: Bootstrap starting.
I20260925 22:49:17.408686 1268 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 64617aec2aef48b883194cc873b75a2f: Neither blocks nor log segments found. Creating new log.
I20260925 22:49:17.408915 1347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:17.396252 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.198.60:39493 (local address 127.0.0.1:45080)
0925 22:49:17.396531 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0925 22:49:17.396547 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 22:49:17.396629 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:49:17.397367 (+ 738us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:49:17.397371 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:49:17.397563 (+ 192us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:49:17.397682 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:49:17.397690 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:49:17.398821 (+ 1131us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:49:17.398825 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:49:17.399661 (+ 836us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:49:17.399677 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:49:17.400296 (+ 619us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:49:17.400300 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:49:17.400385 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:49:17.400877 (+ 492us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:49:17.400892 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:49:17.404225 (+ 3333us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:49:17.406940 (+ 2715us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:49:17.406947 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:49:17.406950 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:49:17.407238 (+ 288us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:49:17.407608 (+ 370us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:49:17.407614 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:49:17.407619 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:49:17.407748 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:49:17.408214 (+ 466us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:49:17.408218 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:49:17.408364 (+ 146us) client_negotiation.cc:769] Sending connection context
0925 22:49:17.408483 (+ 119us) client_negotiation.cc:240] Negotiation successful
0925 22:49:17.408676 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":190}
I20260925 22:49:17.409623 1268 log.cc:824] T 00000000000000000000000000000000 P 64617aec2aef48b883194cc873b75a2f: Log is configured to *not* fsync() on all Append() calls
I20260925 22:49:17.409821 1424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:17.396780 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45080 (local address 127.10.198.60:39493)
0925 22:49:17.397123 (+ 343us) server_negotiation.cc:206] Beginning negotiation
0925 22:49:17.397132 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 22:49:17.397144 (+ 12us) server_negotiation.cc:407] Connection header received
0925 22:49:17.397176 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:49:17.397179 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:49:17.397221 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:49:17.397794 (+ 573us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:49:17.397924 (+ 130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:49:17.398684 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:49:17.399828 (+ 1144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:49:17.400162 (+ 334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:49:17.400292 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:49:17.404896 (+ 4604us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:49:17.404900 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:49:17.404903 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:49:17.404927 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:49:17.406775 (+ 1848us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:49:17.407377 (+ 602us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:49:17.407381 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:49:17.407383 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:49:17.407503 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:49:17.407887 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:49:17.407893 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:49:17.407897 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:49:17.408102 (+ 205us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:49:17.408223 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0925 22:49:17.409501 (+ 1278us) server_negotiation.cc:299] Negotiation successful
0925 22:49:17.409630 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":4,"server-negotiator.queue_time_us":274,"thread_start_us":150,"threads_started":1}
Sep 25 22:49:17 dist-test-slave-5vdr krb5kdc[1187](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376556, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.198.62@KRBTEST.COM
I20260925 22:49:17.412839 1268 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 64617aec2aef48b883194cc873b75a2f: No bootstrap required, opened a new log
I20260925 22:49:17.413813 1343 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6206349d187a4ed491c2b1822bb6cd16: Bootstrap starting.
Sep 25 22:49:17 dist-test-slave-5vdr krb5kdc[1187](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376557, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.198.60@KRBTEST.COM for kudu/127.10.198.61@KRBTEST.COM
I20260925 22:49:17.416695 1271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:17.403002 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44902 (local address 127.10.198.62:43067)
0925 22:49:17.403237 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0925 22:49:17.403243 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 22:49:17.403939 (+ 696us) server_negotiation.cc:407] Connection header received
0925 22:49:17.404118 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:49:17.404121 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:49:17.404171 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:49:17.404221 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:49:17.405078 (+ 857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:49:17.405790 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:49:17.406820 (+ 1030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:49:17.407067 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:49:17.407181 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:49:17.410776 (+ 3595us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:49:17.410783 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:49:17.410790 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:49:17.410823 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:49:17.413205 (+ 2382us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:49:17.413801 (+ 596us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:49:17.413804 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:49:17.413806 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:49:17.413904 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:49:17.414262 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:49:17.414277 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:49:17.414282 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:49:17.414533 (+ 251us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:49:17.414725 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0925 22:49:17.416356 (+ 1631us) server_negotiation.cc:299] Negotiation successful
0925 22:49:17.416511 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":154,"thread_start_us":112,"threads_started":1}
I20260925 22:49:17.417930 1343 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6206349d187a4ed491c2b1822bb6cd16: Neither blocks nor log segments found. Creating new log.
I20260925 22:49:17.417748 1268 raft_consensus.cc:348] T 00000000000000000000000000000000 P 64617aec2aef48b883194cc873b75a2f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64617aec2aef48b883194cc873b75a2f" member_type: VOTER last_known_addr { host: "127.10.198.62" port: 43067 } } peers { permanent_uuid: "6206349d187a4ed491c2b1822bb6cd16" member_type: VOTER last_known_addr { host: "127.10.198.61" port: 41425 } } peers { permanent_uuid: "00ac2a44184548daa6c6ecd21ddacbe4" member_type: VOTER last_known_addr { host: "127.10.198.60" port: 39493 } }
I20260925 22:49:17.418064 1268 raft_consensus.cc:729] T 00000000000000000000000000000000 P 64617aec2aef48b883194cc873b75a2f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 64617aec2aef48b883194cc873b75a2f, State: Initialized, Role: FOLLOWER
I20260925 22:49:17.418430 1268 consensus_queue.cc:260] T 00000000000000000000000000000000 P 64617aec2aef48b883194cc873b75a2f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64617aec2aef48b883194cc873b75a2f" member_type: VOTER last_known_addr { host: "127.10.198.62" port: 43067 } } peers { permanent_uuid: "6206349d187a4ed491c2b1822bb6cd16" member_type: VOTER last_known_addr { host: "127.10.198.61" port: 41425 } } peers { permanent_uuid: "00ac2a44184548daa6c6ecd21ddacbe4" member_type: VOTER last_known_addr { host: "127.10.198.60" port: 39493 } }
I20260925 22:49:17.418928 1343 log.cc:824] T 00000000000000000000000000000000 P 6206349d187a4ed491c2b1822bb6cd16: Log is configured to *not* fsync() on all Append() calls
I20260925 22:49:17.419348 1427 sys_catalog.cc:455] T 00000000000000000000000000000000 P 64617aec2aef48b883194cc873b75a2f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64617aec2aef48b883194cc873b75a2f" member_type: VOTER last_known_addr { host: "127.10.198.62" port: 43067 } } peers { permanent_uuid: "6206349d187a4ed491c2b1822bb6cd16" member_type: VOTER last_known_addr { host: "127.10.198.61" port: 41425 } } peers { permanent_uuid: "00ac2a44184548daa6c6ecd21ddacbe4" member_type: VOTER last_known_addr { host: "127.10.198.60" port: 39493 } } }
I20260925 22:49:17.419486 1427 sys_catalog.cc:458] T 00000000000000000000000000000000 P 64617aec2aef48b883194cc873b75a2f [sys.catalog]: This master's current role is: FOLLOWER
I20260925 22:49:17.420543 1268 sys_catalog.cc:564] T 00000000000000000000000000000000 P 64617aec2aef48b883194cc873b75a2f [sys.catalog]: configured and running, proceeding with master startup.
I20260925 22:49:17.420624 1423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:17.410330 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.198.61:41425 (local address 127.0.0.1:39944)
0925 22:49:17.410495 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0925 22:49:17.410507 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 22:49:17.410594 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:49:17.410959 (+ 365us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:49:17.410962 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:49:17.411152 (+ 190us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:49:17.411294 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:49:17.411303 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:49:17.412338 (+ 1035us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:49:17.412342 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:49:17.413129 (+ 787us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:49:17.413141 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:49:17.413851 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:49:17.413855 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:49:17.413952 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:49:17.414815 (+ 863us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:49:17.414830 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:49:17.416401 (+ 1571us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:49:17.418667 (+ 2266us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:49:17.418671 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:49:17.418676 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:49:17.418949 (+ 273us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:49:17.419328 (+ 379us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:49:17.419334 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:49:17.419339 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:49:17.419441 (+ 102us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:49:17.420030 (+ 589us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:49:17.420034 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:49:17.420147 (+ 113us) client_negotiation.cc:769] Sending connection context
0925 22:49:17.420244 (+ 97us) client_negotiation.cc:240] Negotiation successful
0925 22:49:17.420419 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":94}
I20260925 22:49:17.421208 1346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:17.410419 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39944 (local address 127.10.198.61:41425)
0925 22:49:17.410645 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0925 22:49:17.410651 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 22:49:17.410666 (+ 15us) server_negotiation.cc:407] Connection header received
0925 22:49:17.410705 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:49:17.410710 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:49:17.410758 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:49:17.410829 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:49:17.411437 (+ 608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:49:17.412177 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:49:17.413290 (+ 1113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:49:17.413608 (+ 318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:49:17.413710 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:49:17.416559 (+ 2849us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:49:17.416564 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:49:17.416567 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:49:17.416595 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:49:17.418516 (+ 1921us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:49:17.419076 (+ 560us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:49:17.419080 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:49:17.419081 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:49:17.419170 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:49:17.419566 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:49:17.419571 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:49:17.419574 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:49:17.419904 (+ 330us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:49:17.420845 (+ 941us) server_negotiation.cc:1035] Waiting for connection context
0925 22:49:17.420933 (+ 88us) server_negotiation.cc:299] Negotiation successful
0925 22:49:17.421067 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":146}
I20260925 22:49:17.421978 1418 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.198.60" port: 39493 } has no permanent_uuid. Determining permanent_uuid...
I20260925 22:49:17.423553 1343 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6206349d187a4ed491c2b1822bb6cd16: No bootstrap required, opened a new log
Sep 25 22:49:17 dist-test-slave-5vdr krb5kdc[1187](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376556, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.198.61@KRBTEST.COM
Sep 25 22:49:17 dist-test-slave-5vdr krb5kdc[1187](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790376557, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.198.60@KRBTEST.COM for kudu/127.10.198.60@KRBTEST.COM
I20260925 22:49:17.428169 1343 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6206349d187a4ed491c2b1822bb6cd16 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64617aec2aef48b883194cc873b75a2f" member_type: VOTER last_known_addr { host: "127.10.198.62" port: 43067 } } peers { permanent_uuid: "6206349d187a4ed491c2b1822bb6cd16" member_type: VOTER last_known_addr { host: "127.10.198.61" port: 41425 } } peers { permanent_uuid: "00ac2a44184548daa6c6ecd21ddacbe4" member_type: VOTER last_known_addr { host: "127.10.198.60" port: 39493 } }
I20260925 22:49:17.428431 1343 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6206349d187a4ed491c2b1822bb6cd16 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6206349d187a4ed491c2b1822bb6cd16, State: Initialized, Role: FOLLOWER
I20260925 22:49:17.428822 1343 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6206349d187a4ed491c2b1822bb6cd16 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64617aec2aef48b883194cc873b75a2f" member_type: VOTER last_known_addr { host: "127.10.198.62" port: 43067 } } peers { permanent_uuid: "6206349d187a4ed491c2b1822bb6cd16" member_type: VOTER last_known_addr { host: "127.10.198.61" port: 41425 } } peers { permanent_uuid: "00ac2a44184548daa6c6ecd21ddacbe4" member_type: VOTER last_known_addr { host: "127.10.198.60" port: 39493 } }
I20260925 22:49:17.430496 1343 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6206349d187a4ed491c2b1822bb6cd16 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 22:49:17.426894 1200 master_runner.cc:186] Error getting master registration for 127.10.198.61:41425: OK, instance_id { permanent_uuid: "6206349d187a4ed491c2b1822bb6cd16" instance_seqno: 1790376557132157 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260925 22:49:17.431020 1200 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260925 22:49:17.431921 1428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:17.418666 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39956 (local address 127.10.198.61:41425)
0925 22:49:17.419753 (+ 1087us) server_negotiation.cc:206] Beginning negotiation
0925 22:49:17.419762 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 22:49:17.419775 (+ 13us) server_negotiation.cc:407] Connection header received
0925 22:49:17.419809 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:49:17.419811 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:49:17.419899 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:49:17.420957 (+ 1058us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:49:17.421071 (+ 114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:49:17.421802 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:49:17.422918 (+ 1116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:49:17.423291 (+ 373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:49:17.423375 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:49:17.427021 (+ 3646us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:49:17.427028 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:49:17.427033 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:49:17.427060 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:49:17.428970 (+ 1910us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:49:17.430173 (+ 1203us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:49:17.430177 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:49:17.430178 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:49:17.430277 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:49:17.430625 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:49:17.430630 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:49:17.430634 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:49:17.430910 (+ 276us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:49:17.431110 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0925 22:49:17.431505 (+ 395us) server_negotiation.cc:299] Negotiation successful
0925 22:49:17.431648 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1007,"thread_start_us":140,"threads_started":1}
I20260925 22:49:17.432189 1423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:17.422554 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.198.60:39493 (local address 127.0.0.1:45086)
0925 22:49:17.422868 (+ 314us) negotiation.cc:107] Waiting for socket to connect
0925 22:49:17.422886 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0925 22:49:17.422971 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 22:49:17.423233 (+ 262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 22:49:17.423237 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 22:49:17.423481 (+ 244us) client_negotiation.cc:189] Negotiated authn=SASL
0925 22:49:17.423580 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:49:17.423587 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:49:17.424648 (+ 1061us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:49:17.424655 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:49:17.425548 (+ 893us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 22:49:17.425563 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 22:49:17.426284 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 22:49:17.426289 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 22:49:17.426376 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:49:17.426872 (+ 496us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 22:49:17.426892 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0925 22:49:17.428619 (+ 1727us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 22:49:17.430656 (+ 2037us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:49:17.430658 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:49:17.430660 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:49:17.430874 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:49:17.431162 (+ 288us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 22:49:17.431165 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 22:49:17.431167 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 22:49:17.431247 (+ 80us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 22:49:17.431680 (+ 433us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 22:49:17.431682 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 22:49:17.431759 (+ 77us) client_negotiation.cc:769] Sending connection context
0925 22:49:17.431884 (+ 125us) client_negotiation.cc:240] Negotiation successful
0925 22:49:17.432039 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":230}
I20260925 22:49:17.432773 1424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:17.422742 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45086 (local address 127.10.198.60:39493)
0925 22:49:17.422868 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0925 22:49:17.422874 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 22:49:17.422928 (+ 54us) server_negotiation.cc:407] Connection header received
0925 22:49:17.423066 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:49:17.423072 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:49:17.423131 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:49:17.423218 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:49:17.423693 (+ 475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:49:17.424517 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:49:17.425675 (+ 1158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:49:17.426049 (+ 374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:49:17.426152 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:49:17.428787 (+ 2635us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:49:17.428793 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:49:17.428799 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:49:17.428828 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:49:17.430545 (+ 1717us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:49:17.430987 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:49:17.430990 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:49:17.430991 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:49:17.431065 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:49:17.431360 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:49:17.431362 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:49:17.431364 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:49:17.431562 (+ 198us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:49:17.431648 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0925 22:49:17.432512 (+ 864us) server_negotiation.cc:299] Negotiation successful
0925 22:49:17.432632 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260925 22:49:17.436012 11032 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
W20260925 22:49:17.436007 1440 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 64617aec2aef48b883194cc873b75a2f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
I20260925 22:49:17.436218 1418 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 00ac2a44184548daa6c6ecd21ddacbe4: Bootstrap starting.
W20260925 22:49:17.436371 1440 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 22:49:17.437315 1242 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "64617aec2aef48b883194cc873b75a2f"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44902
I20260925 22:49:17.437551 1242 raft_consensus.cc:482] T 00000000000000000000000000000000 P 64617aec2aef48b883194cc873b75a2f [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260925 22:49:17.437665 1242 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 64617aec2aef48b883194cc873b75a2f [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:49:17.438890 1418 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 00ac2a44184548daa6c6ecd21ddacbe4: Neither blocks nor log segments found. Creating new log.
I20260925 22:49:17.439112 1242 raft_consensus.cc:504] T 00000000000000000000000000000000 P 64617aec2aef48b883194cc873b75a2f [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64617aec2aef48b883194cc873b75a2f" member_type: VOTER last_known_addr { host: "127.10.198.62" port: 43067 } } peers { permanent_uuid: "6206349d187a4ed491c2b1822bb6cd16" member_type: VOTER last_known_addr { host: "127.10.198.61" port: 41425 } } peers { permanent_uuid: "00ac2a44184548daa6c6ecd21ddacbe4" member_type: VOTER last_known_addr { host: "127.10.198.60" port: 39493 } }
I20260925 22:49:17.439751 1418 log.cc:824] T 00000000000000000000000000000000 P 00ac2a44184548daa6c6ecd21ddacbe4: Log is configured to *not* fsync() on all Append() calls
I20260925 22:49:17.441035 1317 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "64617aec2aef48b883194cc873b75a2f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "6206349d187a4ed491c2b1822bb6cd16"
I20260925 22:49:17.441321 1317 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6206349d187a4ed491c2b1822bb6cd16 [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:49:17.442387 1418 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 00ac2a44184548daa6c6ecd21ddacbe4: No bootstrap required, opened a new log
I20260925 22:49:17.442770 1317 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 6206349d187a4ed491c2b1822bb6cd16 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 64617aec2aef48b883194cc873b75a2f in term 1.
I20260925 22:49:17.444223 1441 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6206349d187a4ed491c2b1822bb6cd16 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64617aec2aef48b883194cc873b75a2f" member_type: VOTER last_known_addr { host: "127.10.198.62" port: 43067 } } peers { permanent_uuid: "6206349d187a4ed491c2b1822bb6cd16" member_type: VOTER last_known_addr { host: "127.10.198.61" port: 41425 } } peers { permanent_uuid: "00ac2a44184548daa6c6ecd21ddacbe4" member_type: VOTER last_known_addr { host: "127.10.198.60" port: 39493 } } }
I20260925 22:49:17.444396 1441 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6206349d187a4ed491c2b1822bb6cd16 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 22:49:17.441720 1275 master_runner.cc:186] Error getting master registration for 127.10.198.60:39493: OK, instance_id { permanent_uuid: "00ac2a44184548daa6c6ecd21ddacbe4" instance_seqno: 1790376557374468 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260925 22:49:17.445202 1275 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260925 22:49:17.445621 1218 leader_election.cc:302] T 00000000000000000000000000000000 P 64617aec2aef48b883194cc873b75a2f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6206349d187a4ed491c2b1822bb6cd16, 64617aec2aef48b883194cc873b75a2f; no voters:
I20260925 22:49:17.446008 1427 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 64617aec2aef48b883194cc873b75a2f [term 1 FOLLOWER]: Leader election won for term 1
I20260925 22:49:17.446265 1427 raft_consensus.cc:686] T 00000000000000000000000000000000 P 64617aec2aef48b883194cc873b75a2f [term 1 LEADER]: Becoming Leader. State: Replica: 64617aec2aef48b883194cc873b75a2f, State: Running, Role: LEADER
I20260925 22:49:17.446710 1427 consensus_queue.cc:237] T 00000000000000000000000000000000 P 64617aec2aef48b883194cc873b75a2f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64617aec2aef48b883194cc873b75a2f" member_type: VOTER last_known_addr { host: "127.10.198.62" port: 43067 } } peers { permanent_uuid: "6206349d187a4ed491c2b1822bb6cd16" member_type: VOTER last_known_addr { host: "127.10.198.61" port: 41425 } } peers { permanent_uuid: "00ac2a44184548daa6c6ecd21ddacbe4" member_type: VOTER last_known_addr { host: "127.10.198.60" port: 39493 } }
I20260925 22:49:17.448096 1418 raft_consensus.cc:348] T 00000000000000000000000000000000 P 00ac2a44184548daa6c6ecd21ddacbe4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64617aec2aef48b883194cc873b75a2f" member_type: VOTER last_known_addr { host: "127.10.198.62" port: 43067 } } peers { permanent_uuid: "6206349d187a4ed491c2b1822bb6cd16" member_type: VOTER last_known_addr { host: "127.10.198.61" port: 41425 } } peers { permanent_uuid: "00ac2a44184548daa6c6ecd21ddacbe4" member_type: VOTER last_known_addr { host: "127.10.198.60" port: 39493 } }
I20260925 22:49:17.448448 1418 raft_consensus.cc:729] T 00000000000000000000000000000000 P 00ac2a44184548daa6c6ecd21ddacbe4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00ac2a44184548daa6c6ecd21ddacbe4, State: Initialized, Role: FOLLOWER
W20260925 22:49:17.448659 1452 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 6206349d187a4ed491c2b1822bb6cd16: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 22:49:17.448740 1452 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 22:49:17.448854 1427 sys_catalog.cc:455] T 00000000000000000000000000000000 P 64617aec2aef48b883194cc873b75a2f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 64617aec2aef48b883194cc873b75a2f. Latest consensus state: current_term: 1 leader_uuid: "64617aec2aef48b883194cc873b75a2f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64617aec2aef48b883194cc873b75a2f" member_type: VOTER last_known_addr { host: "127.10.198.62" port: 43067 } } peers { permanent_uuid: "6206349d187a4ed491c2b1822bb6cd16" member_type: VOTER last_known_addr { host: "127.10.198.61" port: 41425 } } peers { permanent_uuid: "00ac2a44184548daa6c6ecd21ddacbe4" member_type: VOTER last_known_addr { host: "127.10.198.60" port: 39493 } } }
I20260925 22:49:17.449013 1427 sys_catalog.cc:458] T 00000000000000000000000000000000 P 64617aec2aef48b883194cc873b75a2f [sys.catalog]: This master's current role is: LEADER
I20260925 22:49:17.448930 1418 consensus_queue.cc:260] T 00000000000000000000000000000000 P 00ac2a44184548daa6c6ecd21ddacbe4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64617aec2aef48b883194cc873b75a2f" member_type: VOTER last_known_addr { host: "127.10.198.62" port: 43067 } } peers { permanent_uuid: "6206349d187a4ed491c2b1822bb6cd16" member_type: VOTER last_known_addr { host: "127.10.198.61" port: 41425 } } peers { permanent_uuid: "00ac2a44184548daa6c6ecd21ddacbe4" member_type: VOTER last_known_addr { host: "127.10.198.60" port: 39493 } }
I20260925 22:49:17.449447 1458 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 22:49:17.449994 1242 leader_election.cc:288] T 00000000000000000000000000000000 P 64617aec2aef48b883194cc873b75a2f [CANDIDATE]: Term 1 election: Requested vote from peers 6206349d187a4ed491c2b1822bb6cd16 (127.10.198.61:41425), 00ac2a44184548daa6c6ecd21ddacbe4 (127.10.198.60:39493)
I20260925 22:49:17.450507 1418 sys_catalog.cc:564] T 00000000000000000000000000000000 P 00ac2a44184548daa6c6ecd21ddacbe4 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 22:49:17.450670 1458 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 22:49:17.451975 1392 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "64617aec2aef48b883194cc873b75a2f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "00ac2a44184548daa6c6ecd21ddacbe4"
I20260925 22:49:17.452286 1392 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 00ac2a44184548daa6c6ecd21ddacbe4 [term 0 FOLLOWER]: Advancing to term 1
I20260925 22:49:17.453886 1392 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 00ac2a44184548daa6c6ecd21ddacbe4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 64617aec2aef48b883194cc873b75a2f in term 1.
I20260925 22:49:17.456652 1392 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 00ac2a44184548daa6c6ecd21ddacbe4 [term 1 FOLLOWER]: Refusing update from remote peer 64617aec2aef48b883194cc873b75a2f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 22:49:17.456966 1317 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 6206349d187a4ed491c2b1822bb6cd16 [term 1 FOLLOWER]: Refusing update from remote peer 64617aec2aef48b883194cc873b75a2f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 22:49:17.457840 1477 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 64617aec2aef48b883194cc873b75a2f [LEADER]: Connected to new peer: Peer: permanent_uuid: "00ac2a44184548daa6c6ecd21ddacbe4" member_type: VOTER last_known_addr { host: "127.10.198.60" port: 39493 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 22:49:17.458118 1427 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 64617aec2aef48b883194cc873b75a2f [LEADER]: Connected to new peer: Peer: permanent_uuid: "6206349d187a4ed491c2b1822bb6cd16" member_type: VOTER last_known_addr { host: "127.10.198.61" port: 41425 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 22:49:17.463655 1441 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6206349d187a4ed491c2b1822bb6cd16 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 64617aec2aef48b883194cc873b75a2f. Latest consensus state: current_term: 1 leader_uuid: "64617aec2aef48b883194cc873b75a2f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64617aec2aef48b883194cc873b75a2f" member_type: VOTER last_known_addr { host: "127.10.198.62" port: 43067 } } peers { permanent_uuid: "6206349d187a4ed491c2b1822bb6cd16" member_type: VOTER last_known_addr { host: "127.10.198.61" port: 41425 } } peers { permanent_uuid: "00ac2a44184548daa6c6ecd21ddacbe4" member_type: VOTER last_known_addr { host: "127.10.198.60" port: 39493 } } }
I20260925 22:49:17.463898 1441 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6206349d187a4ed491c2b1822bb6cd16 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 22:49:17.464018 1478 sys_catalog.cc:455] T 00000000000000000000000000000000 P 00ac2a44184548daa6c6ecd21ddacbe4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "64617aec2aef48b883194cc873b75a2f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64617aec2aef48b883194cc873b75a2f" member_type: VOTER last_known_addr { host: "127.10.198.62" port: 43067 } } peers { permanent_uuid: "6206349d187a4ed491c2b1822bb6cd16" member_type: VOTER last_known_addr { host: "127.10.198.61" port: 41425 } } peers { permanent_uuid: "00ac2a44184548daa6c6ecd21ddacbe4" member_type: VOTER last_known_addr { host: "127.10.198.60" port: 39493 } } }
I20260925 22:49:17.464161 1478 sys_catalog.cc:458] T 00000000000000000000000000000000 P 00ac2a44184548daa6c6ecd21ddacbe4 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 22:49:17.464255 1478 sys_catalog.cc:455] T 00000000000000000000000000000000 P 00ac2a44184548daa6c6ecd21ddacbe4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 64617aec2aef48b883194cc873b75a2f. Latest consensus state: current_term: 1 leader_uuid: "64617aec2aef48b883194cc873b75a2f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64617aec2aef48b883194cc873b75a2f" member_type: VOTER last_known_addr { host: "127.10.198.62" port: 43067 } } peers { permanent_uuid: "6206349d187a4ed491c2b1822bb6cd16" member_type: VOTER last_known_addr { host: "127.10.198.61" port: 41425 } } peers { permanent_uuid: "00ac2a44184548daa6c6ecd21ddacbe4" member_type: VOTER last_known_addr { host: "127.10.198.60" port: 39493 } } }
I20260925 22:49:17.464355 1478 sys_catalog.cc:458] T 00000000000000000000000000000000 P 00ac2a44184548daa6c6ecd21ddacbe4 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 22:49:17.466912 1472 mvcc.cc:205] Tried to move back new op lower bound from 7333382379335458816 to 7333382379306082304. Current Snapshot: MvccSnapshot[applied={T|T < 7333382379335458816}]
I20260925 22:49:17.467391 1427 sys_catalog.cc:455] T 00000000000000000000000000000000 P 64617aec2aef48b883194cc873b75a2f [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "64617aec2aef48b883194cc873b75a2f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64617aec2aef48b883194cc873b75a2f" member_type: VOTER last_known_addr { host: "127.10.198.62" port: 43067 } } peers { permanent_uuid: "6206349d187a4ed491c2b1822bb6cd16" member_type: VOTER last_known_addr { host: "127.10.198.61" port: 41425 } } peers { permanent_uuid: "00ac2a44184548daa6c6ecd21ddacbe4" member_type: VOTER last_known_addr { host: "127.10.198.60" port: 39493 } } }
I20260925 22:49:17.467545 1427 sys_catalog.cc:458] T 00000000000000000000000000000000 P 64617aec2aef48b883194cc873b75a2f [sys.catalog]: This master's current role is: LEADER
I20260925 22:49:17.468835 1478 sys_catalog.cc:455] T 00000000000000000000000000000000 P 00ac2a44184548daa6c6ecd21ddacbe4 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "64617aec2aef48b883194cc873b75a2f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64617aec2aef48b883194cc873b75a2f" member_type: VOTER last_known_addr { host: "127.10.198.62" port: 43067 } } peers { permanent_uuid: "6206349d187a4ed491c2b1822bb6cd16" member_type: VOTER last_known_addr { host: "127.10.198.61" port: 41425 } } peers { permanent_uuid: "00ac2a44184548daa6c6ecd21ddacbe4" member_type: VOTER last_known_addr { host: "127.10.198.60" port: 39493 } } }
I20260925 22:49:17.468972 1478 sys_catalog.cc:458] T 00000000000000000000000000000000 P 00ac2a44184548daa6c6ecd21ddacbe4 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 22:49:17.470083 1458 catalog_manager.cc:1347] Generated new cluster ID: 50d4d33013964b669802a9bfd88886df
I20260925 22:49:17.470149 1458 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 22:49:17.473659 1441 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6206349d187a4ed491c2b1822bb6cd16 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "64617aec2aef48b883194cc873b75a2f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64617aec2aef48b883194cc873b75a2f" member_type: VOTER last_known_addr { host: "127.10.198.62" port: 43067 } } peers { permanent_uuid: "6206349d187a4ed491c2b1822bb6cd16" member_type: VOTER last_known_addr { host: "127.10.198.61" port: 41425 } } peers { permanent_uuid: "00ac2a44184548daa6c6ecd21ddacbe4" member_type: VOTER last_known_addr { host: "127.10.198.60" port: 39493 } } }
I20260925 22:49:17.473825 1441 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6206349d187a4ed491c2b1822bb6cd16 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 22:49:17.474594 1427 sys_catalog.cc:455] T 00000000000000000000000000000000 P 64617aec2aef48b883194cc873b75a2f [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "64617aec2aef48b883194cc873b75a2f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64617aec2aef48b883194cc873b75a2f" member_type: VOTER last_known_addr { host: "127.10.198.62" port: 43067 } } peers { permanent_uuid: "6206349d187a4ed491c2b1822bb6cd16" member_type: VOTER last_known_addr { host: "127.10.198.61" port: 41425 } } peers { permanent_uuid: "00ac2a44184548daa6c6ecd21ddacbe4" member_type: VOTER last_known_addr { host: "127.10.198.60" port: 39493 } } }
I20260925 22:49:17.474728 1427 sys_catalog.cc:458] T 00000000000000000000000000000000 P 64617aec2aef48b883194cc873b75a2f [sys.catalog]: This master's current role is: LEADER
I20260925 22:49:17.475915 1480 mvcc.cc:205] Tried to move back new op lower bound from 7333382379335458816 to 7333382379306082304. Current Snapshot: MvccSnapshot[applied={T|T < 7333382379335458816}]
W20260925 22:49:17.478057 1476 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 00ac2a44184548daa6c6ecd21ddacbe4: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 22:49:17.478152 1476 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 22:49:17.487064 1271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:17.455610 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44914 (local address 127.10.198.62:43067)
0925 22:49:17.455792 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0925 22:49:17.455799 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 22:49:17.467001 (+ 11202us) server_negotiation.cc:407] Connection header received
0925 22:49:17.467123 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:49:17.467127 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:49:17.467194 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:49:17.467284 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:49:17.468590 (+ 1306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:49:17.469387 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:49:17.470665 (+ 1278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:49:17.471045 (+ 380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:49:17.471128 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:49:17.482661 (+ 11533us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:49:17.482666 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:49:17.482669 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:49:17.482718 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:49:17.484816 (+ 2098us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:49:17.485479 (+ 663us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:49:17.485483 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:49:17.485486 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:49:17.485593 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:49:17.485930 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:49:17.485934 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:49:17.485936 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:49:17.486166 (+ 230us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:49:17.486307 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0925 22:49:17.486751 (+ 444us) server_negotiation.cc:299] Negotiation successful
0925 22:49:17.486897 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260925 22:49:17.491698 1458 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 22:49:17.493198 1458 catalog_manager.cc:1504] Loading token signing keys...
I20260925 22:49:17.508234 1458 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 64617aec2aef48b883194cc873b75a2f: Generated new TSK 0
I20260925 22:49:17.509310 1458 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 22:49:17.517004 1428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:17.466202 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39972 (local address 127.10.198.61:41425)
0925 22:49:17.466344 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0925 22:49:17.466350 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 22:49:17.472239 (+ 5889us) server_negotiation.cc:407] Connection header received
0925 22:49:17.476968 (+ 4729us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:49:17.476972 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:49:17.477030 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:49:17.477085 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:49:17.480340 (+ 3255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:49:17.481234 (+ 894us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:49:17.500042 (+ 18808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:49:17.500511 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:49:17.500625 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:49:17.512131 (+ 11506us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:49:17.512136 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:49:17.512139 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:49:17.512174 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:49:17.514174 (+ 2000us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:49:17.514806 (+ 632us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:49:17.514812 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:49:17.514814 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:49:17.514930 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:49:17.515338 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:49:17.515344 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:49:17.515347 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:49:17.515625 (+ 278us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:49:17.515797 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0925 22:49:17.516640 (+ 843us) server_negotiation.cc:299] Negotiation successful
0925 22:49:17.516811 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260925 22:49:17.521334 1424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:17.466677 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45090 (local address 127.10.198.60:39493)
0925 22:49:17.466807 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0925 22:49:17.466812 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:49:17.472462 (+ 5650us) server_negotiation.cc:407] Connection header received
0925 22:49:17.475791 (+ 3329us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:49:17.475795 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:49:17.475896 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:49:17.475959 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:49:17.478012 (+ 2053us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:49:17.478907 (+ 895us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:49:17.500848 (+ 21941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:49:17.501245 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:49:17.501334 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:49:17.505611 (+ 4277us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 22:49:17.505615 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 22:49:17.505621 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 22:49:17.505659 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0925 22:49:17.507437 (+ 1778us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:49:17.510685 (+ 3248us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:49:17.510690 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:49:17.510691 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:49:17.510805 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 22:49:17.511167 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 22:49:17.511170 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 22:49:17.511172 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 22:49:17.520228 (+ 9056us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 22:49:17.520448 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0925 22:49:17.520939 (+ 491us) server_negotiation.cc:299] Negotiation successful
0925 22:49:17.521110 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
W20260925 22:49:17.564268 1428 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:49:17.560436 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39988 (local address 127.10.198.61:41425)
0925 22:49:17.560609 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0925 22:49:17.560614 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:49:17.562816 (+ 2202us) server_negotiation.cc:407] Connection header received
0925 22:49:17.563047 (+ 231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:49:17.563051 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:49:17.563107 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:49:17.563167 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:49:17.564090 (+ 923us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39988: BlockingRecv error: recv got EOF from 127.0.0.1:39988 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
W20260925 22:49:17.566743 1424 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:49:17.561578 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45104 (local address 127.10.198.60:39493)
0925 22:49:17.561710 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0925 22:49:17.561716 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 22:49:17.565422 (+ 3706us) server_negotiation.cc:407] Connection header received
0925 22:49:17.565734 (+ 312us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:49:17.565739 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:49:17.565799 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:49:17.565861 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:49:17.566543 (+ 682us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45104: BlockingRecv error: recv got EOF from 127.0.0.1:45104 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
W20260925 22:49:17.568329 1271 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:49:17.560699 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44928 (local address 127.10.198.62:43067)
0925 22:49:17.560833 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0925 22:49:17.560839 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 22:49:17.565191 (+ 4352us) server_negotiation.cc:407] Connection header received
0925 22:49:17.567293 (+ 2102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:49:17.567298 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:49:17.567373 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:49:17.567439 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:49:17.568144 (+ 705us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44928: BlockingRecv error: recv got EOF from 127.0.0.1:44928 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
W20260925 22:49:17.574503 1428 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:49:17.572417 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39996 (local address 127.10.198.61:41425)
0925 22:49:17.572563 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0925 22:49:17.572568 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:49:17.573202 (+ 634us) server_negotiation.cc:407] Connection header received
0925 22:49:17.573514 (+ 312us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:49:17.573518 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:49:17.573615 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:49:17.573678 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:49:17.574357 (+ 679us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39996: BlockingRecv error: recv got EOF from 127.0.0.1:39996 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
W20260925 22:49:17.581210 1428 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:49:17.579317 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40002 (local address 127.10.198.61:41425)
0925 22:49:17.579481 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0925 22:49:17.579486 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:49:17.579950 (+ 464us) server_negotiation.cc:407] Connection header received
0925 22:49:17.580228 (+ 278us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:49:17.580233 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:49:17.580301 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:49:17.580369 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:49:17.581069 (+ 700us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40002: BlockingRecv error: recv got EOF from 127.0.0.1:40002 (error 108)
Metrics: {"server-negotiator.queue_time_us":77}
W20260925 22:49:17.588768 1428 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:49:17.586570 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40006 (local address 127.10.198.61:41425)
0925 22:49:17.586735 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0925 22:49:17.586741 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 22:49:17.587468 (+ 727us) server_negotiation.cc:407] Connection header received
0925 22:49:17.587726 (+ 258us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:49:17.587730 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:49:17.587798 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:49:17.587924 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:49:17.588604 (+ 680us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40006: BlockingRecv error: recv got EOF from 127.0.0.1:40006 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
W20260925 22:49:17.598206 1428 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:49:17.596094 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40018 (local address 127.10.198.61:41425)
0925 22:49:17.596301 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0925 22:49:17.596307 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 22:49:17.597038 (+ 731us) server_negotiation.cc:407] Connection header received
0925 22:49:17.597299 (+ 261us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:49:17.597303 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:49:17.597368 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:49:17.597428 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:49:17.598075 (+ 647us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40018: BlockingRecv error: recv got EOF from 127.0.0.1:40018 (error 108)
Metrics: {"server-negotiator.queue_time_us":100}
W20260925 22:49:17.611426 1428 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:49:17.609427 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40030 (local address 127.10.198.61:41425)
0925 22:49:17.609557 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0925 22:49:17.609562 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:49:17.610263 (+ 701us) server_negotiation.cc:407] Connection header received
0925 22:49:17.610487 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:49:17.610490 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:49:17.610550 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:49:17.610606 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:49:17.611286 (+ 680us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40030: BlockingRecv error: recv got EOF from 127.0.0.1:40030 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
W20260925 22:49:17.633076 1428 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:49:17.630661 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40042 (local address 127.10.198.61:41425)
0925 22:49:17.630838 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0925 22:49:17.630843 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:49:17.631675 (+ 832us) server_negotiation.cc:407] Connection header received
0925 22:49:17.632109 (+ 434us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:49:17.632113 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:49:17.632171 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:49:17.632226 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:49:17.632946 (+ 720us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40042: BlockingRecv error: recv got EOF from 127.0.0.1:40042 (error 108)
Metrics: {"server-negotiator.queue_time_us":77}
W20260925 22:49:17.670392 1428 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:49:17.668348 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40054 (local address 127.10.198.61:41425)
0925 22:49:17.668491 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0925 22:49:17.668496 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:49:17.669229 (+ 733us) server_negotiation.cc:407] Connection header received
0925 22:49:17.669479 (+ 250us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:49:17.669483 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:49:17.669545 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:49:17.669606 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:49:17.670258 (+ 652us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40054: BlockingRecv error: recv got EOF from 127.0.0.1:40054 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
W20260925 22:49:17.738346 1428 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:49:17.737258 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40070 (local address 127.10.198.61:41425)
0925 22:49:17.737442 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0925 22:49:17.737446 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:49:17.737527 (+ 81us) server_negotiation.cc:407] Connection header received
0925 22:49:17.737651 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:49:17.737654 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:49:17.737704 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:49:17.737794 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:49:17.738223 (+ 429us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40070: BlockingRecv error: recv got EOF from 127.0.0.1:40070 (error 108)
Metrics: {"server-negotiator.queue_time_us":92}
W20260925 22:49:17.870443 1428 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:49:17.869017 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40084 (local address 127.10.198.61:41425)
0925 22:49:17.869216 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0925 22:49:17.869222 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 22:49:17.869406 (+ 184us) server_negotiation.cc:407] Connection header received
0925 22:49:17.869597 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:49:17.869600 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:49:17.869651 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:49:17.869756 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:49:17.870289 (+ 533us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40084: BlockingRecv error: recv got EOF from 127.0.0.1:40084 (error 108)
Metrics: {"server-negotiator.queue_time_us":88}
W20260925 22:49:18.130199 1428 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 22:49:18.128892 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40100 (local address 127.10.198.61:41425)
0925 22:49:18.129054 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0925 22:49:18.129059 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:49:18.129212 (+ 153us) server_negotiation.cc:407] Connection header received
0925 22:49:18.129402 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:49:18.129405 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:49:18.129459 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:49:18.129557 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0925 22:49:18.130051 (+ 494us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40100: BlockingRecv error: recv got EOF from 127.0.0.1:40100 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
I20260925 22:49:18.450376 1452 catalog_manager.cc:1259] Loaded cluster ID: 50d4d33013964b669802a9bfd88886df
I20260925 22:49:18.450459 1452 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 6206349d187a4ed491c2b1822bb6cd16: loading cluster ID for follower catalog manager: success
I20260925 22:49:18.452661 1452 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 6206349d187a4ed491c2b1822bb6cd16: acquiring CA information for follower catalog manager: success
I20260925 22:49:18.454356 1452 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 6206349d187a4ed491c2b1822bb6cd16: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260925 22:49:18.479518 1476 catalog_manager.cc:1259] Loaded cluster ID: 50d4d33013964b669802a9bfd88886df
I20260925 22:49:18.479605 1476 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 00ac2a44184548daa6c6ecd21ddacbe4: loading cluster ID for follower catalog manager: success
I20260925 22:49:18.481485 1476 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 00ac2a44184548daa6c6ecd21ddacbe4: acquiring CA information for follower catalog manager: success
I20260925 22:49:18.482837 1476 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 00ac2a44184548daa6c6ecd21ddacbe4: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260925 22:49:18.648937 1555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:18.644723 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40104 (local address 127.10.198.61:41425)
0925 22:49:18.644967 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0925 22:49:18.644971 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 22:49:18.645015 (+ 44us) server_negotiation.cc:407] Connection header received
0925 22:49:18.645163 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:49:18.645166 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:49:18.645297 (+ 131us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:49:18.645411 (+ 114us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 22:49:18.645677 (+ 266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:49:18.646225 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:49:18.647188 (+ 963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:49:18.647420 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:49:18.647502 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:49:18.648008 (+ 506us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 22:49:18.648485 (+ 477us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 22:49:18.648589 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0925 22:49:18.648721 (+ 132us) server_negotiation.cc:299] Negotiation successful
0925 22:49:18.648801 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":169,"thread_start_us":94,"threads_started":1}
I20260925 22:49:18.656646 1561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:18.652536 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45114 (local address 127.10.198.60:39493)
0925 22:49:18.652810 (+ 274us) server_negotiation.cc:206] Beginning negotiation
0925 22:49:18.652813 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 22:49:18.652901 (+ 88us) server_negotiation.cc:407] Connection header received
0925 22:49:18.653116 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:49:18.653119 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:49:18.653245 (+ 126us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:49:18.653386 (+ 141us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 22:49:18.653596 (+ 210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:49:18.654127 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:49:18.655056 (+ 929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:49:18.655276 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:49:18.655365 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:49:18.655641 (+ 276us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 22:49:18.656175 (+ 534us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 22:49:18.656272 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0925 22:49:18.656366 (+ 94us) server_negotiation.cc:299] Negotiation successful
0925 22:49:18.656425 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":196,"thread_start_us":82,"threads_started":1}
I20260925 22:49:18.664449 1561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:18.660694 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45126 (local address 127.10.198.60:39493)
0925 22:49:18.660852 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0925 22:49:18.660857 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:49:18.661007 (+ 150us) server_negotiation.cc:407] Connection header received
0925 22:49:18.661197 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:49:18.661201 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:49:18.661258 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:49:18.661344 (+ 86us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 22:49:18.661587 (+ 243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:49:18.662102 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:49:18.663176 (+ 1074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:49:18.663437 (+ 261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:49:18.663578 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:49:18.663884 (+ 306us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 22:49:18.663980 (+ 96us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 22:49:18.664065 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0925 22:49:18.664201 (+ 136us) server_negotiation.cc:299] Negotiation successful
0925 22:49:18.664276 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260925 22:49:18.665295 1555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:18.660281 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40118 (local address 127.10.198.61:41425)
0925 22:49:18.660426 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0925 22:49:18.660431 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:49:18.660649 (+ 218us) server_negotiation.cc:407] Connection header received
0925 22:49:18.660846 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:49:18.660849 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:49:18.660924 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:49:18.660970 (+ 46us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 22:49:18.661345 (+ 375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:49:18.662108 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:49:18.663667 (+ 1559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:49:18.664058 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:49:18.664146 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:49:18.664746 (+ 600us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 22:49:18.664840 (+ 94us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 22:49:18.664890 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0925 22:49:18.665065 (+ 175us) server_negotiation.cc:299] Negotiation successful
0925 22:49:18.665144 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260925 22:49:18.666119 1568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 22:49:18.660058 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44936 (local address 127.10.198.62:43067)
0925 22:49:18.660408 (+ 350us) server_negotiation.cc:206] Beginning negotiation
0925 22:49:18.660413 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 22:49:18.660428 (+ 15us) server_negotiation.cc:407] Connection header received
0925 22:49:18.660630 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 22:49:18.660637 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 22:49:18.660829 (+ 192us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 22:49:18.660947 (+ 118us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 22:49:18.661223 (+ 276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:49:18.662096 (+ 873us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:49:18.663666 (+ 1570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 22:49:18.664061 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 22:49:18.664160 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 22:49:18.664562 (+ 402us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 22:49:18.665084 (+ 522us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 22:49:18.665902 (+ 818us) server_negotiation.cc:1035] Waiting for connection context
0925 22:49:18.665937 (+ 35us) server_negotiation.cc:299] Negotiation successful
0925 22:49:18.666003 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":263,"thread_start_us":72,"threads_started":1}
I20260925 22:49:18.669692 11032 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrWXm8X/build/debug/bin/kudu with pid 1200
I20260925 22:49:18.676292 11032 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrWXm8X/build/debug/bin/kudu with pid 1275
I20260925 22:49:18.682979 11032 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrWXm8X/build/debug/bin/kudu with pid 1349
2026-09-25T22:49:18Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (3178 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (6562 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (299022 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST